Prototype network feature reconstruction and classification method, device and equipment

Abstract

The application provides a prototype network feature reconstruction and classification method, device and equipment, comprising: obtaining N prototypes of a prototype network; determining the Euclidean distance between the feature vectors of the i-th prototype in any two prototypes is less than a preset threshold; determining the to-be-inserted feature vector in the prototype according to K feature vectors in the two prototypes; inserting the to-be-inserted feature vector into the set position of the two prototypes respectively, reconstructing the feature of the to-be-identified object through the above method, performing three spline interpolations on the two prototypes after feature reconstruction and the feature of the to-be-identified object after reconstruction, and deriving the two prototypes and the feature of the to-be-identified object after three spline interpolations, and determining the category of the to-be-identified object according to the similarity between the derivative of the feature of the to-be-identified object and the derivative of the two prototypes. The problem that the prototypes in the prototype network are too far away from the target and cause classification errors is solved, and the classification accuracy is improved.

Description

Technical Field

[0001] This application relates to the field of image classification technology, specifically to a method, apparatus, and device for feature reconstruction and classification of a prototype network. Background Technology

[0002] In few-sample classification problems, one of the most pressing issues to address is overfitting. Due to the limited amount of data, general classification algorithms tend to overfit, leading to significant discrepancies between the classification results and the actual data. To mitigate the impact of overfitting caused by insufficient data, metric-based meta-learning methods can be employed, and prototype networks are generated based on such a method.

[0003] Prototype networks calculate prototypes based on training set feature data. However, prototypes may be too close together, potentially leading to classification errors in image classification. To avoid this, existing solutions propose adding a set of parameters to increase the distance between different prototypes, thereby improving classification accuracy. However, this method is complex and time-consuming, requiring significant computing resources. Summary of the Invention

[0004] To address the issue of classification errors caused by the close proximity of prototypes in prototype networks, a feature reconstruction and classification method, apparatus, and device for prototype networks are proposed. This method is simpler and requires less time and computer resources.

[0005] Firstly, this application provides a feature reconstruction method for a prototype network, the method comprising:

[0006] Obtain N prototypes of the prototype network. Different prototypes are used to identify different object categories. Each prototype includes M ordered feature vectors; M and N are both positive integers.

[0007] The Euclidean distance between the i-th feature vector of the first prototype and the i-th feature vector of the second prototype is less than a preset threshold. The first prototype and the second prototype are two prototypes among the N prototypes.

[0008] A first feature vector to be inserted in the first prototype is determined based on K feature vectors in the first prototype; the distance between the index of the K feature vectors in the first prototype and the index of the i-th feature vector is less than a set value; and a second feature vector to be inserted in the second prototype is determined based on K feature vectors in the second prototype; the distance between the index of the K feature vectors in the second prototype and the index of the i-th feature vector is less than a set value.

[0009] The first feature vector to be inserted is inserted into the set position of the first prototype, and the second feature vector to be inserted is inserted into the set position of the second prototype.

[0010] In one possible implementation, the feature vector to be inserted in the third prototype is determined based on K feature vectors in the third prototype; the distance between the index of the K feature vectors in the third prototype and the index of the i-th feature vector in the third prototype is less than a set value; the third prototype is any prototype other than the first prototype and the second prototype among the N prototypes.

[0011] The feature vector to be inserted in the third prototype is inserted into the set position in the third prototype.

[0012] In one possible implementation, the set position is located between the i-th feature vector and the (i+1)-th feature vector; or, the set position is located between the (i-1)-th feature vector and the i-th feature vector.

[0013] Secondly, this application provides a classification method based on prototype networks, the method comprising:

[0014] A prototype network is used to extract feature points of the target object in the image to be detected. The feature points include M sequentially arranged feature vectors; M and N are both positive integers.

[0015] Obtain N prototypes in the prototype network reconstructed based on the method described in the first aspect above, and the positions where feature vectors are inserted in the reconstructed N prototypes;

[0016] Based on the K feature vectors in the feature point, determine the feature vector to be inserted in the feature point, insert the feature vector to be inserted into the position, and obtain the reconstructed feature point of the target object. The distance between the index of the K feature vectors in the feature point and the index of the feature vector corresponding to the position is less than a set value.

[0017] Calculate the Euclidean distance between the reconstructed feature points and the N reconstructed prototypes, and determine the category of the target object based on the Euclidean distance.

[0018] In one possible approach, the Euclidean distance between the reconstructed feature points and the N reconstructed prototypes is calculated, and the category of the target object is determined based on the Euclidean distance, including:

[0019] A function that constructs the reconstructed feature points using the index of the feature vector as the x-axis and the eigenvalue of the feature vector as the y-axis, and a function that reconstructs the N prototypes;

[0020] Obtain the first rate of change of the function of the reconstructed feature points, and obtain the second rate of change of the function of the N prototypes obtained from the reconstruction respectively;

[0021] The category of the target object is determined based on the comparison results between the rate of change of the first function and N rates of change of the second function.

[0022] In one possible implementation, determining the category of the target object based on comparisons between the first function's rate of change and N rates of change of the second function includes:

[0023] The comparison result between the rate of change of the first function and N rates of change of the second function is determined by the following formula:

[0024] arctan(the rate of change of the m-th second function) - arctan(the rate of change of the first function);

[0025] When the value of the comparison result is greater than a first preset value and less than a second preset value, and the arctan (the m-th rate of change of the second function) has the same sign as the arctan (the rate of change of the first function), the target object belongs to the m-th class.

[0026] Thirdly, this application provides a feature reconstruction apparatus for a prototype network, the apparatus comprising:

[0027] The acquisition module acquires N prototypes from the prototype network. Different prototypes are used to identify different object categories. Each prototype includes M ordered feature vectors; M and N are both positive integers.

[0028] The distance determination module is used to determine that the Euclidean distance between the i-th feature vector of the first prototype and the i-th feature vector of the second prototype is less than a preset threshold, wherein the first prototype and the second prototype are two prototypes among the N prototypes;

[0029] The module for determining the feature vector to be inserted is used to determine a first feature vector to be inserted in the first prototype based on K feature vectors in the first prototype, wherein the distance between the index of the K feature vectors in the first prototype and the index of the i-th feature vector is less than a set value; and to determine a second feature vector to be inserted in the second prototype based on K feature vectors in the second prototype, wherein the distance between the index of the K feature vectors in the second prototype and the index of the i-th feature vector is less than a set value;

[0030] The reconstruction module is used to insert the first feature vector to be inserted into a set position of the first prototype, and to insert the second feature vector to be inserted into the set position of the second prototype.

[0031] Fourthly, this application provides a classification device based on a prototype network, the device comprising:

[0032] The extraction module is used to extract feature points of target objects in the image to be detected using a prototype network. The feature points include M sequentially arranged feature vectors; M and N are both positive integers.

[0033] The acquisition module is used to acquire N prototypes reconstructed from the prototype network based on the method of the first aspect mentioned above, and the positions where feature vectors are inserted in the N reconstructed prototypes;

[0034] The reconstruction module is used to determine the feature vector to be inserted in the feature point based on K feature vectors in the feature point, insert the feature vector to be inserted into the position, and obtain the reconstructed feature point of the target object. The distance between the index of the K feature vectors in the feature point and the index of the feature vector corresponding to the position is less than a set value.

[0035] The classification module is used to calculate the Euclidean distance between the reconstructed feature points and the N reconstructed prototypes, and to determine the category of the target object based on the Euclidean distance.

[0036] Fifthly, this application provides a feature reconstruction and classification device for a prototype network, the device comprising:

[0037] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the methods of the first and second aspects described above.

[0038] In a sixth aspect, this application provides a computer storage medium storing a computer program for causing a computer to perform the methods described in the first and second aspects above.

[0039] This application addresses the problem of target misclassification caused by excessive distance between the prototype and the target object in the prototype network by reconstructing features from the prototype and the prototype in the prototype network. Furthermore, the feature reconstruction process in this application involves inserting feature vectors into the prototype and feature points, eliminating the need for other methods to extract additional vectors, making it simpler. Attached Figure Description

[0040] Figure 1 A schematic diagram of a prototype network principle as exemplified by an exemplary embodiment of the present invention;

[0041] Figure 2This is a schematic diagram illustrating a feature reconstruction method for a prototype network according to an exemplary embodiment of the present invention.

[0042] Figure 3 This is a schematic diagram illustrating a feature reconstruction method for a prototype network according to an exemplary embodiment of the present invention.

[0043] Figure 4 This is a schematic diagram illustrating the mapping of a prototype S1 to a coordinate system according to an exemplary embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram illustrating the mapping of a prototype S2 to a coordinate system according to an exemplary embodiment of the present invention;

[0045] Figure 6 This is a schematic diagram of a prototype S1 interpolation according to an exemplary embodiment of the present invention;

[0046] Figure 7 This is a schematic diagram of a prototype S2 interpolation according to an exemplary embodiment of the present invention;

[0047] Figure 8 This is a schematic diagram of a prototype S1 interpolation sequence number according to an exemplary embodiment of the present invention;

[0048] Figure 9 This is a schematic diagram of a prototype S2 interpolation sequence number according to an exemplary embodiment of the present invention;

[0049] Figure 10 This is a schematic flowchart illustrating a classification method based on a prototype network, as exemplified by an exemplary embodiment of the present invention.

[0050] Figure 11 This is a schematic diagram illustrating another classification method based on a prototype network, as exemplified by an exemplary embodiment of the present invention.

[0051] Figure 12 This is a schematic diagram illustrating a process for comparing the rate of change of a first function with N rates of change of a second function, as exemplified by an exemplary embodiment of the present invention.

[0052] Figure 13 This is a schematic diagram illustrating the mapping of a target object to a coordinate system according to an exemplary embodiment of the present invention;

[0053] Figure 14 This is a schematic diagram of target object interpolation as an example of an exemplary embodiment of the present invention;

[0054] Figure 15 This is a schematic diagram illustrating the function corresponding to the target object in an example of an exemplary embodiment of the present invention;

[0055] Figure 16This is a schematic diagram of the function corresponding to the reconstructed prototype S1' according to an exemplary embodiment of the present invention;

[0056] Figure 17 This is a schematic diagram of the function corresponding to the reconstructed prototype S2' according to an exemplary embodiment of the present invention;

[0057] Figure 18 This is a schematic diagram of a feature reconstruction apparatus for a prototype network according to an exemplary embodiment of the present invention;

[0058] Figure 19 This is a schematic diagram of a prototype network-based classification device according to an exemplary embodiment of the present invention;

[0059] Figure 20 This is a schematic diagram of a prototype network feature reconstruction and classification device as an example of an exemplary embodiment of the present invention. Detailed Implementation

[0060] The technical solutions in the embodiments of this application will now be described clearly and in detail with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0061] First, let's introduce the prototype network:

[0062] For classification problems with small sample sizes, a metric-based meta-learning method can be used. The prototype network projects the samples into a metric space (or feature space). Each sample can be projected into a point in this feature space (called a feature point or feature vector). Then, the category to which the sample belongs is determined by measuring the distance (e.g., Euclidean distance) between the sample and the cluster center corresponding to the classification category of each image in the metric feature space.

[0063] According to metric-based meta-learning, the two key points of metric learning are: 1. How to project samples onto a feature space; 2. How to describe the position of a class in the feature space, so that unlabeled samples can calculate their distances to each class. Prototype networks are generated based on these two key points; they are parameterized neural networks that project samples onto a feature space.

[0064] The prototype network will now be explained in conjunction with two key points of meta-learning of the aforementioned metrics. First, let's address the first key point: the prototype network can be represented by the function f. θ (x) represents the function f. θ(x) can be understood as the process of projecting samples onto the feature space, where x represents the feature vector of the sample, the function value represents the value of each sample after being projected onto that feature space (also called a feature point; a sample can be projected into a feature point in the feature space), and θ represents the parameters of the prototype network. The parameters θ need to be learned and can be considered to determine the position of the feature point corresponding to each sample in the feature space. Therefore, the essence of the first key point is how to make the prototype network f θ (x) learns a good parameter θ value so that samples of the same category are projected to feature points in the feature space that are closer together.

[0065] Secondly, let's explain the second key point: Taking images as the training data for the prototype network, each set of training data is divided into a support set and a query set. The support set consists of some labeled samples. For example, if there are N image categories and M samples in each category, then the support set is N-way--M-shot. The query set also consists of some labeled samples. The categories of the samples in the query set need to be consistent with the categories of the samples in the support set. For example, if the support set has three categories: "cat", "sofa", and "dog", then the categories of the samples in the query set must also be "cat", "sofa", and "dog". However, the number of samples in each category in the query set can be arbitrary and does not need to be consistent with the number of samples in each category in the support set. In this case, the query set can be denoted as N-way--Q-shot, where Q and M can be the same or different, which is not limited here. Each category is determined by the cluster center of the feature points in the feature space, which is the sum of the positions of all samples in the sample set that belong to this category (e.g., the average of the positions of the feature points in the feature space of all samples in the same category of the same image can be used as the cluster center of that category). This cluster center is the prototype corresponding to each category.

[0066] For ease of understanding, Figure 1For example, consider a 3-way-5-shot sample set, meaning there are 3 categories (ways) and 5 samples (shots) in each category. x1 is a feature point projected onto the feature space from a sample in the query set (this is for illustration only; other samples in the query set are not shown). C1, C2, and C3 are the prototypes corresponding to the 3 categories in the sample set. The 5 points surrounding each prototype are the 5 feature points projected onto the feature space from the 5 samples in the corresponding category. For example, the position of C1 in the feature space is obtained by averaging the positions of the 5 feature points projected onto the feature space from the 5 samples in the corresponding category; the position of C2 is obtained by averaging the positions of the 5 feature points projected onto the feature space from the 5 samples in the corresponding category; and the position of C3 is obtained by averaging the positions of the 5 feature points projected onto the feature space from the 5 samples in the corresponding category. In other words, this feature space contains samples from 3 categories, and samples of the same category are relatively close to each other. To classify sample x1, sample x1 is projected onto this space and the class prototype of which class x1 is closer to is calculated, and then sample x1 is considered to belong to that class.

[0067] Since the class of sample x1 is actually known during training (the class of each sample in the query set is known, i.e., supervised learning), the prototype network f is obtained by comparing the predicted class of sample x1 with its true class using stochastic gradient descent. θ The prototype network f finds the minimum value of the loss function (x) and thus learns a good value of θ after convergence. Therefore, it can be considered that after training, this prototype network f... θ (x) can project samples of the same category into a region that is relatively close to each other. In practical applications, for samples y of unknown category that are not labeled, this prototype network can be used to project the sample y onto the feature space and calculate which category's prototype the sample y is closest to. For example, if it is closest to prototype C1, then the category of sample y is considered to be the category corresponding to C1.

[0068] However, in practical applications, the prototype network described above encounters difficulties when handling samples of similar categories. For example, if a sample's feature points projected onto the feature space through the prototype network are nearly identical to, or even indistinguishable from, the prototypes of the two categories,... Figure 1 In the sample x1, the distances from C2 and C3 are basically the same. The prototype network has difficulty classifying this sample x1, which will greatly limit the classification accuracy of the prototype network.

[0069] To address the aforementioned problems, embodiments of this application provide a feature reconstruction method for a prototype network, such as... Figure 2 As shown, the method includes:

[0070] S201: Obtain N prototypes of the prototype network. Different prototypes are used to identify different object categories. Each prototype includes M ordered feature vectors; M and N are both positive integers.

[0071] Taking images as an example, the prototype network is first used to extract feature points for each image. These feature points include M sequentially arranged feature vectors. For example, a feature point might be fθ1 = [a, a1, a2], which includes three feature vectors: a, a1, and a2. The index of a is 0, a1 is 1, and a2 is 2. The prototype for each category is calculated based on multiple feature points within that category. Specifically, the average value of the positions of the feature points projected onto the feature space of each image within the same category can be used as the prototype for that category. The calculation method is standard knowledge in this field and will not be elaborated here. Since the prototype is obtained from the image's feature points, the feature points and the prototype have the same dimension.

[0072] The training set images are input into the prototype network to obtain the prototypes corresponding to each category. During the classification process, the Euclidean distance between the feature points of the image to be detected and the prototypes of each category is calculated. The category corresponding to the prototype with the closest distance is taken as the category of the image to be detected.

[0073] S202: Determine that the Euclidean distance between the i-th feature vector included in the first prototype and the i-th feature vector included in the second prototype is less than a preset threshold, wherein the first prototype and the second prototype are two prototypes among the N prototypes.

[0074] This step is to determine the insertion position of the feature vector to be inserted. Taking any two prototypes out of the N prototypes of the prototype network as an example, the Euclidean distance of the feature vectors is calculated using the following formula 1:

[0075] |y(s 1,i )-y(s 2,i )|

[0076] Where s 1,i and s 2,i Let y(s) represent the indices of the i-th vector of the first prototype and the i-th vector of the second prototype, respectively. 1,i ) and y(s 2,i ) respectively represent s 1,i and s 2,i If the Euclidean distance of the corresponding feature vector is less than the preset distance threshold as determined by the above formula, it indicates that the i-th feature vector in prototype S1 and the i-th feature vector in prototype S2 are close. If feature reconstruction is not performed, the subsequent detection object may be classified incorrectly due to the close distance between the two prototypes.

[0077] Once the proximity of feature vectors among the feature points is determined, the insertion position of the feature vector to be inserted can also be determined. In one possible implementation, the insertion position of the feature vector to be inserted is located between the i-th feature vector and the (i+1)-th feature vector; or between the (i-1)-th feature vector and the i-th feature vector.

[0078] It is important to note that when calculating the Euclidean distance between eigenvectors of two feature points, the Euclidean distance is calculated between eigenvectors with the same index. Specifically, the Euclidean distance is calculated between two eigenvectors in prototype S1 with index 0, and between two eigenvectors in prototype S1 with index 1. The eigenvector indices can start from 0 or 1; this is not specifically limited in this embodiment.

[0079] In addition, to ensure that the dimensions of each reconstructed prototype are the same, after determining the insertion position of the feature vector to be inserted in prototype S1, the feature vector to be inserted is inserted at the same position in the other prototypes.

[0080] S203: Determine a first feature vector to be inserted in the first prototype based on the K feature vectors in the first prototype; the distance between the index of the K feature vectors in the first prototype and the index of the i-th feature vector is less than a set value; and determine a second feature vector to be inserted in the second prototype based on the K feature vectors in the second prototype; the distance between the index of the K feature vectors in the second prototype and the index of the i-th feature vector is less than a set value.

[0081] After determining the insertion position of the vector to be inserted, the eigenvalues ​​of the eigenvector to be inserted are determined as follows:

[0082] A method similar to Gaussian blur is used to interpolate neighboring points. The Gaussian kernel used in this method is shown below:

[0083]

[0084] First, rewrite the Gaussian kernel as an 8*1 vector: [0.095, 0.118, 0.075, 0.118, 0.118, 0.095, 0.118, 0.095]. Taking the eigenvector with index i in prototype S1 as an example (s... 1,i Then, the eigenvalues ​​of the feature vector to be inserted are estimated using the following formula 2:

[0085] y(s 1,i )=y(s 1,i -K)×0.095+y(s 1,i -(K-1))×0.118+y(s1,i -(K-2))×0.075+y(s 1,i -(K-3))×0.118+y(s 1,i +(K-3))×0.118+y(s 1,i +(K-2))×0.095+y(s 1,i +(K-1))×0.118+y(s 1,i +K)×0.095

[0086] Where K represents the maximum distance between the indices of the K feature vectors and the indices of the i-th feature vector. In the embodiment provided in this application, the i-th feature vector is used as the reference, and four feature vectors before and after it are taken to calculate the feature value of the vector to be inserted using the above formula. Alternatively, the i-th feature vector can be used as the reference, and five feature vectors before and after it can be taken to calculate the feature value of the vector to be inserted using the above formula, or only the eight feature vectors after the reference vector can be used for calculation; no specific limitation is made here. If s 1,i The 'i' in the model corresponds to a smaller index, for example, the feature vector with index 0 in prototype S1. In this case, there are no feature vectors preceding this feature vector. y(s 1,0 -K), y(s) 1,0 -(K-1)), y(s 1,0 -(K-2)), y(s 1,0 If -(K-3)) does not exist, then y(s) 1,0 -K), y(s) 1,0 -(K-1)), y(s 1,0 -(K-2)), y(s 1,0 -(K-3)) is set to 0; for example, prototype S1 includes 8 feature vectors, s 1,i If i is 7, and there are no other eigenvectors following this eigenvector, then y(s) 1,0 +K)), y(s 1,0 +(K-1)), y(s 1,0 +(K-2)), y(s 1,0 If +(K-3)) does not exist, then y(s) 1,0 +K)), y(s 1,0 +(K-1)), y(s 1,0 +(K-2)), y(s 1,0 +K-3) is set to 0.

[0087] In one possible implementation, the third prototype, other than the first and second prototypes, is operated on as follows, specifically as follows: Figure 3 As shown:

[0088] S301: Determine the feature vector to be inserted in the third prototype based on the K feature vectors in the third prototype; the distance between the index of the K feature vectors in the third prototype and the index of the i-th feature vector in the third prototype is less than a set value; the third prototype is any prototype other than the first prototype and the second prototype among the N prototypes.

[0089] S302: Insert the feature vector to be inserted in the third prototype into the set position in the third prototype.

[0090] Once the insertion position of the vector to be inserted in a certain prototype is determined, the corresponding feature vector is inserted at the corresponding position in the remaining prototypes to ensure that the dimensions of each prototype are the same. Specifically, the eigenvalue of the feature vector to be inserted can be calculated using Formula 2 above. It is also important to note that only one feature vector is inserted between two feature vectors. For example, if the Euclidean distance between the feature vector with index 0 in prototype S1 and the feature vector with index 0 in prototype S2 is less than a preset threshold, then a vector needs to be inserted between the feature vector with index 0 and the feature vector with index 1 in both prototypes S1 and S2. Similarly, a feature vector is inserted between the feature vector with index 0 and the feature vector with index 1 in the remaining prototypes, and its eigenvalue is calculated according to Formula 2 above. If the Euclidean distance between the feature vector with index 0 in prototype S1 and the feature vector with index 0 in prototype S3 is less than a preset threshold, since the eigenvalue of the feature vector to be inserted calculated according to Formula 2 above will not change, it is not necessary to insert the same eigenvalue at the same position. Furthermore, if it is determined that a vector is inserted between the feature vector with index 0 and the feature vector with index 1 in prototype S1 and prototype S2, there is no need to judge whether the Euclidean distance between the feature vector with index 0 and the feature vector with index 1 in the other prototypes is less than a preset threshold, and a feature vector can be directly inserted.

[0091] S204: Insert the first feature vector to be inserted into the set position of the first prototype, and insert the second feature vector to be inserted into the set position of the second prototype.

[0092] The position set in S204 is the same as the insertion position of the feature vector to be inserted in S202 and S203.

[0093] The following section provides a detailed description of the implementation methods for feature reconstruction of prototype networks.

[0094] Taking image feature reconstruction as an example, if the dataset includes two types of images, "smiling faces" and "crying faces," and each type of image includes two images (two smiling face images and two crying face images), by extracting features from each image in each type, we can obtain the feature points fθ corresponding to each image. 11 fθ 12 fθ21 fθ 22 Using feature points fθ 11 fθ 12 The prototype S1 = [a a1 a2 a3 a4 a5 a6 a7] corresponding to the "smiley face" category is calculated, and the feature point fθ is used. 21 fθ 22 The prototype S2 corresponding to the "crying face" category is calculated as [b b1 b2 b3 b4 b5 b6 b7], where both prototype S1 and prototype S2 include 7 feature vectors.

[0095] In one possible implementation, prototypes S1 and S2 are mapped onto a coordinate system, where the horizontal axis represents the index of each feature sequence and the vertical axis represents the eigenvalue of the feature vector. The coordinate systems of prototypes S1 and S2 are as follows: Figure 4 and Figure 5 As shown.

[0096] Then, using the formula for calculating Euclidean distance in S202, the Euclidean distances between a and b, a1 and b1, a2 and b2, a3 and b3, a4 and b4, a5 and b5, a6 and b6, and a7 and b7 are calculated respectively, resulting in eight Euclidean distances d0, d1, d2, d3, d4, d5, d6, and d7. These distances are then compared with a preset threshold d. Based on the comparison results, if the Euclidean distances between feature vectors a and b, and between feature vectors a2 and b2, are determined to be less than the preset threshold d, then a feature vector a01 is inserted after or before feature vector a in prototypes S1 and S2, and a feature vector a23 is inserted after or before feature vector a2, as detailed below. Figure 6 As shown; insert an eigenvector and eigenvector b01 after or before vector b, and insert an eigenvector b23 after or before b2, specifically as follows. Figure 7 As shown, in this embodiment of the application, a feature vector is inserted after the vector. Figure 6 and Figure 7 The eigenvector is inserted after the eigenvector. The original eigenvector is represented by an empty circle, and the newly inserted eigenvector is represented by "*". Its eigenvalue can be calculated according to formula 2 in S203.

[0097] The positions of the prototypes containing a01, a23, b01, and b23 (the positions set in S204 above) need to be stored. To facilitate storage and retrieval, the newly inserted feature vectors are numbered. To clearly indicate the insertion position, the indices between 0 and 1 are directly set to decimals, such as 1 / 2 and 5 / 2, as detailed below. Figure 8 and Figure 9As shown. Besides the above settings, you can also first set the serial numbers between 0 and 1 to decimals such as 1 / 2 and 5 / 2, then round them down, and shift all subsequent serial numbers one unit to the right. The specific serial number settings can be customized according to different needs; no specific limitations are specified here.

[0098] In one possible implementation, a01, a23, b01, and b23 may not need to be numbered; it is sufficient to save the "position between number 0 and number 1" and the "position between number 2 and number 3".

[0099] After the above feature reconstruction process, the prototype S1' corresponding to the "smiley face" category is [a a01 a1 a2 a23 a3a4 a5 a6 a7], and the prototype S2' corresponding to the "sad face" category is [b b01 b1 b2 b23 b3 b4 b5 b6 b7].

[0100] In the subsequent classification process, the images to be classified are classified according to prototypes S1' and S2'. This application provides a classification method based on prototype networks, such as... Figure 10 As shown:

[0101] S1001: Use a prototype network to extract feature points of the target object in the image to be detected. The feature points include M sequentially arranged feature vectors; M and N are both positive integers.

[0102] The number of feature vectors of the feature points corresponding to the target object in the image to be detected is the same as the number of feature vectors of prototype S1 and prototype S2 in the above feature reconstruction process.

[0103] Before classifying the target object, the prototype network needs to be trained. By comparing the predicted category of the training set image (its feature points and the reconstructed prototype S1' or reconstructed prototype S2') with its true category (labeled), the minimum value of the loss function of the prototype network is found by using the stochastic gradient descent method. After convergence, a good parameter value is learned. Thus, it can be considered that after training, this prototype network can project samples of the same category into a region that is relatively close to each other.

[0104] S1002: Obtain the N prototypes in the prototype network reconstructed based on the methods described in S201-S204 above, and the positions where feature vectors are inserted in the N reconstructed prototypes.

[0105] According to the embodiment of the feature reconstruction method of the above prototype network, the feature points corresponding to the image to be detected are subjected to the same interpolation operation as prototype S1 and prototype S2, wherein the interpolation positions are the positions corresponding to sequence number 1 / 2 and sequence number 5 / 2.

[0106] S1003: Determine the feature vector to be inserted in the feature point based on the K feature vectors in the feature point, insert the feature vector to be inserted into the position, and obtain the reconstructed feature point of the target object. The distance between the index of the K feature vectors in the feature point and the index of the feature vector corresponding to the position is less than a set value.

[0107] Its eigenvalues ​​are calculated using Formula 2 in the above embodiment of the feature reconstruction method for the prototype network, and will not be elaborated here.

[0108] S1004: Calculate the Euclidean distance between the reconstructed feature points and the N prototypes obtained from the reconstruction, and determine the category of the target object based on the Euclidean distance.

[0109] This step yields N Euclidean distances. Then, the softmax algorithm is used to calculate N probabilities between the reconstructed feature points and all the reconstructed N prototypes. If any of these probabilities is greater than a preset threshold, it indicates that the target object belongs to this category.

[0110] If all N probabilities are less than the preset probability threshold, it indicates a classification error. In this case, the following steps are taken, specifically as follows: Figure 11 As shown:

[0111] S1101: A function that constructs the reconstructed feature points using the index of the feature vector as the x-axis and the eigenvalue of the feature vector as the y-axis, and a function that reconstructs the N prototypes;

[0112] S1102: Obtain the first function rate of change of the function of the reconstructed feature points, and obtain the second function rate of change of the functions of the N prototypes obtained from the reconstruction respectively;

[0113] S1103: Determine the category of the target object based on the comparison results of the first function change rate with N second function change rates respectively.

[0114] Through such Figure 12 The steps are as follows: compare the rate of change of the first function with N rates of change of the second function respectively.

[0115] S1201: The comparison results of the first function's rate of change and N second function's rates of change are determined using the following formula 3:

[0116] arctan(the m-th rate of change of the second function) - arctan(the rate of change of the first function);

[0117] S1202: When the value of the comparison result is greater than a first preset value and less than a second preset value, and the arctan (the m-th rate of change of the second function) has the same sign as the arctan (the rate of change of the first function), the target object belongs to the m-th class.

[0118] The following section introduces the specific implementation methods of the classification method based on the aforementioned feature reconstruction methods:

[0119] The image to be detected undergoes feature reconstruction, the specific process of which is described in S1001-S1003 above, and will not be repeated here. Figure 13 As shown, the feature points of the target object in the image to be detected are mapped to the coordinate system to obtain fθ. d =[c c1 c2c3 c4 c5 c6 c7], such as Figure 14 As shown, the reconstructed feature points fθ obtained according to the above feature reconstruction method are... d = [c c01 c1 c2 c23 c3 c4 c5 c6 c7]. By calculating fθ separately... d The Euclidean distances to prototypes S1' and S2' are calculated, and then fθ is calculated using the softmax algorithm. d The probability P1, fθ between the prototype S1' and the prototype S1' d The probability P2 between the target object and the prototype S2' is such that if P1 > the preset probability threshold (0.5), the target object belongs to the "smiley face" category; if P2 > the preset probability threshold (0.5), the target object belongs to the "sad face" category; if both P1 and P2 are greater than the preset probability threshold (0.5), the target object belongs to the category with the higher probability.

[0120] If both P1 and P2 are less than 0.5, the following implementation method shall be adopted:

[0121] like Figure 15 As shown, the feature points fθ after reconstruction are constructed. d The corresponding function yp(s3) specifically includes, but is not limited to, the cubic spline interpolation method. The two prototypes S1' and S2' obtained from the reconstruction are determined, and functions for prototypes S1' and S2' are constructed, that is... Figure 8 and Figure 9 Cubic spline interpolation is performed on the points in the model to transform the discrete points into a continuous function, ultimately yielding the function y1(s1) corresponding to the prototype S1', as shown below. Figure 16 As shown, and the function y2(s2) corresponding to the prototype S2', as follows: Figure 17 As shown.

[0122] Then, respectively, for y pDifferentiating yp(s3), y1(s1), and y2(s2) yields yp'(s3), y1'(s1), and y2'(s2). Then, comparing yp'(s3) and y1'(s1), and yp'(s3) and y2'(s2) respectively, according to the following formulas:

[0123] If |arctan(y1'(s1)-arctan(yp'(s3))|<e, and arctan(y1'(s1)) and arctan(yp'(s3)) have the same sign, then the target object belongs to the "smiley face" class; if arctan(y2'(s2)) and arctan(yp'(s3)) have the same sign, and |arctan(y2'(s2)-arctan(yp'(s3))|<e, then the target object belongs to the "sad face" class, where e is a preset value and can be taken as... If |arctan(y1'(s1)-arctan(yp'(s3))|<e, and arctan(y1'(s1)) and arctan(yp'(s3)) have the same sign, and arctan(y2'(s2)) and arctan(yp'(s3)) have the same sign, and |arctan(y2'(s)-arctan(yp'(s3))|<e, the system will report an error, indicating a classification error.

[0124] By implementing the above-described feature reconstruction and classification methods, the problem of inaccurate classification of target objects caused by the close proximity of two prototypes in the prototype network can be solved, and the overfitting problem can be addressed to some extent.

[0125] Based on the same inventive concept, this application also provides a feature reconstruction device 1800 for a prototype network, such as... Figure 18 As shown, the device includes:

[0126] Module 1801 is used to obtain N prototypes from the prototype network. Different prototypes are used to identify different object categories. Each prototype includes M ordered feature vectors; M and N are both positive integers.

[0127] The distance determination module 1802 is used to determine that the Euclidean distance between the i-th feature vector of the first prototype and the i-th feature vector of the second prototype is less than a preset threshold, wherein the first prototype and the second prototype are two prototypes among the N prototypes;

[0128] The feature vector to be inserted module 1803 is used to determine a first feature vector to be inserted in the first prototype based on K feature vectors in the first prototype; the distance between the index of the K feature vectors in the first prototype and the index of the i-th feature vector is less than a set value; and to determine a second feature vector to be inserted in the second prototype based on K feature vectors in the second prototype; the distance between the index of the K feature vectors in the second prototype and the index of the i-th feature vector is less than a set value;

[0129] The reconstruction module 1804 is used to insert the first feature vector to be inserted into a set position of the first prototype, and to insert the second feature vector to be inserted into the set position of the second prototype.

[0130] In one possible implementation, the reconstruction module 1804 is used to determine the feature vector to be inserted in the third prototype based on K feature vectors in the third prototype; the distance between the index of the K feature vectors in the third prototype and the index of the i-th feature vector in the third prototype is less than a set value; the third prototype is any prototype other than the first prototype and the second prototype among the N prototypes.

[0131] The feature vector to be inserted in the third prototype is inserted into the set position in the third prototype.

[0132] In one possible implementation, the reconstruction module 1804 is used to determine that the set position is located between the i-th feature vector and the (i+1)-th feature vector; or, the set position is located between the (i-1)-th feature vector and the i-th feature vector.

[0133] Based on the same inventive concept, embodiments of this application also provide a classification device 1900 based on a prototype network, such as... Figure 19 As shown, the device includes:

[0134] Extraction module 1901 is used to extract feature points of target objects in the image to be detected using a prototype network. The feature points include M sequentially arranged feature vectors; M and N are both positive integers.

[0135] The acquisition module 1902 is used to acquire N prototypes in the prototype network based on the feature reconstruction method described above, and the positions where feature vectors are inserted in the N reconstructed prototypes;

[0136] The reconstruction module 1903 is used to determine the feature vector to be inserted in the feature point based on K feature vectors in the feature point, insert the feature vector to be inserted into the position, and obtain the reconstructed feature point of the target object. The distance between the index of the K feature vectors in the feature point and the index of the feature vector corresponding to the position is less than a set value.

[0137] The classification module 1904 is used to calculate the Euclidean distance between the reconstructed feature points and the N reconstructed prototypes, and to determine the category of the target object based on the Euclidean distance.

[0138] In one possible implementation, the classification module is used to calculate the Euclidean distance between the reconstructed feature points and the N reconstructed prototypes, and to determine the category of the target object based on the Euclidean distance, including:

[0139] A function that constructs the reconstructed feature points using the index of the feature vector as the x-axis and the eigenvalue of the feature vector as the y-axis, and a function that reconstructs the N prototypes;

[0140] Obtain the first rate of change of the function of the reconstructed feature points, and obtain the second rate of change of the function of the N prototypes obtained from the reconstruction respectively;

[0141] The category of the target object is determined based on the comparison results between the rate of change of the first function and N rates of change of the second function.

[0142] In one possible implementation, the classification module is used to determine the category of the target object based on comparisons between the first function rate of change and N second function rates of change, including:

[0143] The comparison result between the rate of change of the first function and N rates of change of the second function is determined by the following formula:

[0144] arctan(the m-th rate of change of the second function) - arctan(the rate of change of the first function);

[0145] When the value of the comparison result is greater than a first preset value and less than a second preset value, and the arctan (the m-th rate of change of the second function) has the same sign as the arctan (the rate of change of the first function), the target object belongs to the m-th class.

[0146] In one possible implementation, embodiments of this application also provide a feature reconstruction and classification device for a prototype network, such as... Figure 20 As shown, the device includes:

[0147] The device includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the feature reconstruction and classification methods in the above embodiments.

[0148] The following reference Figure 20To describe an electronic device 130 according to this embodiment of the present application. Figure 20 The electronic device 130 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0149] like Figure 20 As shown, the electronic device 130 is presented in the form of a general-purpose electronic device. The components of the electronic device 130 may include, but are not limited to: at least one processor 131, at least one memory 132, and a bus 133 connecting different system components (including memory 132 and processor 131).

[0150] The processor 131 is used to read and execute instructions from the memory 132, so that the at least one processor can execute the feature reconstruction and classification methods provided in the above embodiments.

[0151] Bus 133 represents one or more of several bus structures, including a memory bus or memory controller, peripheral bus, processor, or local bus using any of the various bus structures.

[0152] The memory 132 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 1321 and / or cache memory 1322, and may further include read-only memory (ROM) 1323.

[0153] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0154] Electronic device 130 can also communicate with one or more external devices 134 (e.g., keyboard, pointing device, etc.), and with one or more devices that enable a user to interact with electronic device 130, and / or with any device that enables electronic device 130 to communicate with one or more other electronic devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 135. Furthermore, electronic device 130 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 136. As shown, network adapter 136 communicates with other modules used in electronic device 130 via bus 133. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0155] In some possible implementations, various aspects of the feature reconstruction and classification method provided in this application can also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to cause the computer device to perform the steps of the feature reconstruction and classification method according to various exemplary embodiments of this application described above.

[0156] In addition, this application also provides a computer-readable storage medium storing a computer program for causing a computer to perform the method described in any of the above embodiments.

[0157] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0158] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0159] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0160] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A classification method based on prototype networks, characterized in that, include: The Euclidean distance between the i-th feature vector of the first prototype and the i-th feature vector of the second prototype in any two of the N prototypes of the prototype network is less than a preset threshold. A first feature vector to be inserted in the first prototype is determined based on K feature vectors in the first prototype, and a second feature vector to be inserted in the second prototype is determined based on K feature vectors in the second prototype. Different prototypes are used to identify different object categories. Each prototype includes M feature vectors arranged in order. The distance between the index of the K feature vectors in each prototype and the index of the i-th feature vector is less than a set value. The first feature vector to be inserted is inserted between the i-th feature vector and the (i+1)-th feature vector of the first prototype, or between the (i-1)-th feature vector and the i-th feature vector of the first prototype, and the second feature vector to be inserted is inserted between the i-th feature vector and the (i+1)-th feature vector of the second prototype, or between the (i-1)-th feature vector and the i-th feature vector of the second prototype, to obtain N reconstructed prototypes; A prototype network is used to extract feature points of a target object in an image to be detected. The feature points include M sequentially arranged feature vectors. Based on the K feature vectors in the feature point, determine the feature vector to be inserted in the feature point, insert the feature vector to be inserted into the position of the feature vector in the N prototypes obtained by reconstruction, and obtain the reconstructed feature point of the target object. The distance between the index of the K feature vectors in the feature point and the index of the feature vector corresponding to the position is less than a set value. Calculate the Euclidean distance between the reconstructed feature points and the N reconstructed prototypes, and determine the category of the target object based on the Euclidean distance, where M and N are both positive integers.

2. The method as described in claim 1, characterized in that, Calculate the Euclidean distance between the reconstructed feature points and the N reconstructed prototypes, and determine the category of the target object based on the Euclidean distance, including: A function that constructs the reconstructed feature points using the index of the feature vector as the x-axis and the eigenvalue of the feature vector as the y-axis, and a function that reconstructs the N prototypes; Obtain the first rate of change of the function of the reconstructed feature points, and obtain the second rate of change of the function of the N prototypes obtained from the reconstruction respectively; The category of the target object is determined based on the comparison results between the rate of change of the first function and N rates of change of the second function.

3. The method as described in claim 2, characterized in that, The category of the target object is determined based on the comparison results of the first function's rate of change with N second function's rates of change, including: The comparison result between the rate of change of the first function and N rates of change of the second function is determined by the following formula: ； When the value of the comparison result is greater than a first preset value and less than a second preset value, and the arctan (the m-th rate of change of the second function) has the same sign as the arctan (the rate of change of the first function), the target object belongs to the m-th class.

4. A classification device based on a prototype network, characterized in that, The device includes: The reconstruction module is used to determine that the Euclidean distance between the i-th feature vector of the first prototype and the i-th feature vector of the second prototype is less than a preset threshold when any two prototypes in the N prototypes of the prototype network are N prototypes. The module determines the first feature vector to be inserted in the first prototype based on the K feature vectors in the first prototype, and determines the second feature vector to be inserted in the second prototype based on the K feature vectors in the second prototype. The first feature vector to be inserted is inserted between the i-th feature vector and the (i+1)-th feature vector of the first prototype, or between the (i-1)-th feature vector and the i-th feature vector of the first prototype. Similarly, the second feature vector to be inserted is inserted between the i-th feature vector and the (i+1)-th feature vector of the second prototype, or between the (i-1)-th feature vector and the i-th feature vector of the second prototype, to obtain N reconstructed prototypes. Different prototypes are used to identify different object categories. Each prototype includes M sequentially arranged feature vectors. The distance between the index of K feature vectors and the index of the i-th feature vector in each prototype is less than a set value. The extraction module is used to extract feature points of target objects in the image to be detected using a prototype network. The feature points include M sequentially arranged feature vectors; M and N are both positive integers. The reconstruction module is used to determine the feature vector to be inserted in the feature point based on K feature vectors in the feature point, insert the feature vector to be inserted into the position of the feature vector insertion in the N prototypes obtained by reconstruction, and obtain the reconstructed feature point of the target object. The distance between the index of the K feature vectors in the feature point and the index of the feature vector corresponding to the position is less than a set value. The classification module is used to calculate the Euclidean distance between the reconstructed feature points and the N reconstructed prototypes, and to determine the category of the target object based on the Euclidean distance.

5. A feature classification device for a prototype network, characterized in that, The device includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-3.

6. A computer storage medium, characterized in that, The computer storage medium stores a computer program that enables the computer to perform the method as described in any one of claims 1-3.

Images