A sample labeling method and device

Abstract

The embodiment of the present application provides a sample labeling method and device. This method simultaneously introduces the common features of each label class in the support set, the support set and query set and dynamic features that change based on the labeling task, and the mutual correlation features of the support set label class and the query set in the small sample labeling task, generates a support set label class interactive feature vector and a query set interactive feature vector containing the above features, calculates the similarity between each label class in the query set interactive feature vector and the support set label class interactive feature vector, and uses the label class with the highest similarity to label the samples to be labeled in the query set. The technical solution of the embodiment of the present application makes the support set label class interactive feature vector and the query set interactive feature vector more reasonable in expressing the features of the support set and the query set, is more adaptable to data changes, and can improve the accuracy of sample labeling.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a sample labeling method and device. Background Art

[0002] Due to their high complexity and large number of parameters, deep neural network models often require massive amounts of labeled data for model training, which carries a significant cost. Data labeling can be achieved using annotation models, which also have certain requirements for the size of sample data. For example, when the sample size of images for a certain label (category) is limited, the training of the data annotation model is prone to overfitting to that label, resulting in poor annotation results. Furthermore, considering the time and cost of annotation, as well as the long-tail effect (where a small number of categories contain a large number of samples, while a large number of categories contain only a small number of samples), for specific image annotation tasks, only a small number of image samples can be provided for a given label.

[0003] Therefore, how to use a small amount of labeled data to train a labeling model with excellent classification accuracy and adaptability to dynamic data changes, known as small-sample labeling or small-sample classification, is a key research topic in the industry. However, current small-sample labeling models suffer from problems in practical applications, such as irrational feature extraction and poor adaptability to data changes, making it difficult to improve sample labeling accuracy. Summary of the Invention

[0004] The embodiments of the present application provide a sample labeling method and apparatus to improve the accuracy of sample labeling.

[0005] In a first aspect, an embodiment of the present application provides a sample labeling method, which includes: obtaining a first eigenvector of a support set of a labeling task and a second eigenvector of a query set of a labeling task, the first eigenvector and the second eigenvector containing common features of each label class in the support set; using a convolutional neural network to obtain a third eigenvector of the support set and a fourth eigenvector of the query set, the convolutional neural network containing convolution kernel parameters associated with the features of the support set; calculating the similarity of the query set relative to each label class in the support set based on the first eigenvector and the third eigenvector of the support set and the second eigenvector and the fourth eigenvector of the query set; and labeling the query set based on the similarity.

[0006] The technical solution provided in the embodiment of the present application simultaneously introduces the common features of each label class in the support set, the support set and query set that change according to the features of the support set of the labeling task, and dynamic features, so that the support set and query set can be strengthened in terms of feature expression for the common features of the label class, are more adaptable to data changes, and improve the sample labeling accuracy.

[0007] In one implementation, the common features for each label class in the support set are obtained by average pooling the dimensions corresponding to each label class in the fifth eigenvector of the support set. The fifth eigenvector is obtained by performing an embedding operation on the features of the support set. This average pooling of the dimensions corresponding to each label class in the fifth eigenvector not only obtains the common features for each label class, but also reduces the dimensionality of these common features, which helps reduce the amount of subsequent computation.

[0008] In one implementation, the first eigenvector is obtained by performing vector multiplication and addition on the class scaling factor vector of the support set and the fifth eigenvector; the second eigenvector is obtained by performing vector multiplication and addition on the class scaling factor vector and the sixth eigenvector of the query set; and the sixth eigenvector is obtained by performing an embedding operation on the features of the query set; wherein the class scaling factor vector is obtained by globally pooling the common feature vectors of the common features of each label class in the corresponding support set and then inputting them into a fully connected neural network. In this way, for the current sample labeling task, the embodiment of the present application extracts a class scaling factor vector for characterizing the common features of each label class from the support set input of the current task, and then uses this class scaling factor vector to perform feature enhancement on the support set feature vector and the query set feature vector, so that the difference features between the label classes of the support set and the query set in a single task are amplified, while the similarity features are relatively attenuated, so that the perception of the support set and the query set relative to the current task is enhanced, which is conducive to improving the labeling accuracy. For example, in the task of labeling "football" with "basketball", the color and texture features will be enhanced, and in the task of labeling "basketball" with "flower", the shape features will be enhanced.

[0009] In one implementation, the first eigenvector is obtained by using a normalized exponential function to map each dimension value of the multiplication of the class scale factor vector and the fifth eigenvector to a numerical range of 0 to 1, and then adding the value to the fifth eigenvector. This compresses the common features in the first eigenvector to a numerical range of 0 to 1 without affecting the original expression of the fifth eigenvector.

[0010] In one implementation, the second eigenvector is obtained by using a normalized exponential function to map each dimension value of the multiplication of the class scale factor vector and the sixth eigenvector to a numerical range of 0 to 1, and then adding the value to the sixth eigenvector. This compresses the common features in the second eigenvector to a numerical range of 0 to 1 without affecting the original expression of the sixth eigenvector.

[0011] In one implementation, the convolution kernel parameters are obtained by the following steps: inputting the fifth eigenvector into a convolutional neural network for feature extraction to obtain a task feature vector that represents the overall characteristics of the labeling task; generating a random vector that obeys a multivariate Gaussian distribution, with the random vector having half the dimensional values ​​of the task feature vector as the mean and the other half as the variance; and inputting the random vector into a fully connected neural network for dimensionality reduction to obtain the convolution kernel parameters. It can be seen that the convolution kernel parameters can be dynamically changed as the sample data of the support set changes. Therefore, the third and fourth eigenvectors extracted using such convolution kernel parameters change dynamically as the sample data of the support set and the sample set change, which is beneficial to improving labeling accuracy.

[0012] In one implementation, the similarity of the query set relative to each label class in the support set is calculated based on the first eigenvector and the third eigenvector of the support set and the second eigenvector and the fourth eigenvector of the query set, including: obtaining the seventh eigenvector of the support set and the eighth eigenvector of the query set, the seventh eigenvector is obtained by concatenating the first eigenvector and the third eigenvector, and the eighth eigenvector is obtained by concatenating the second eigenvector and the fourth eigenvector; performing average pooling on the seventh eigenvector to obtain the ninth eigenvector of the support set; obtaining the local correlation eigenvector between the support set and the query set, the local correlation eigenvector is obtained by performing matrix multiplication after dimension conversion on the ninth eigenvector and the eighth eigenvector; obtaining the tenth eigenvector of the support set and the eleventh eigenvector of the query set, the tenth eigenvector is obtained by using the attention mechanism of the ninth eigenvector and the local correlation eigenvector, and the eleventh eigenvector is obtained by using the attention mechanism of the eighth eigenvector and the local correlation eigenvector; determining the similarity of the query set relative to each label class in the support set based on the ninth eigenvector and the tenth eigenvector, and labeling the query set according to the similarity. Currently, common methods generally use convolutional neural network models when generating attention weight distributions, which introduces additional parameters and increases the difficulty of training. However, the embodiments of the present application do not introduce additional parameters when generating attention weight distributions, reducing the training difficulty. In addition, since the embodiments of the present application do not introduce additional parameters, they can be directly transplanted into any pre-trained small sample annotation framework without the need for retraining.

[0013] In one implementation, the tenth eigenvector is obtained by multiplying the ninth eigenvector with the first attention weight distribution vector and then accumulating the ninth eigenvector. The first attention weight distribution vector is obtained by performing horizontal average pooling, normalized exponential function operation and dimensional conversion on the local correlation eigenvector.

[0014] In one implementation, the eleventh eigenvector is obtained by multiplying the eighth eigenvector with the second attention weight distribution vector and then accumulating the eighth eigenvector. The second attention weight distribution vector is obtained by performing vertical average pooling, normalized exponential function operation and dimensional conversion on the local correlation eigenvector.

[0015] In the embodiment of the present application, the local correlation feature vectors are subjected to simple average pooling and softmax operations to obtain respective attention weight distribution vectors, so that the support set and the query set can focus on similar feature parts during subsequent data comparison, thereby helping to improve the labeling accuracy.

[0016] In second aspect, an embodiment of the present application provides a sample labeling device, which includes: a task feature extraction module, used to obtain a first feature vector of a support set of a labeling task and a second feature vector of a query set of the labeling task, the first feature vector and the second feature vector contain common features of each label class in the support set; a dynamic feature extraction module, used to use a convolutional neural network to obtain a third feature vector of the support set and a fourth feature vector of the query set, the convolutional neural network contains convolution kernel parameters associated with the features of the support set; a classification module, used to calculate the similarity of the query set relative to each label class in the support set based on the first feature vector and the third feature vector of the support set and the second feature vector and the fourth feature vector of the query set, and used to label the query set according to the similarity.

[0017] The technical solution provided in the embodiment of the present application simultaneously introduces the common features of each label class in the support set, the support set and query set that change according to the features of the support set of the labeling task, and dynamic features, so that the support set and query set can be strengthened in terms of feature expression for the common features of the label class, are more adaptable to data changes, and improve the sample labeling accuracy.

[0018] In one implementation, the task feature extraction module is further configured to perform average pooling on the dimensions corresponding to each label class in the fifth eigenvector of the support set, where the fifth eigenvector is obtained by performing an embedding operation on the features of the support set, to obtain common features for each label class in the support set. In this way, by performing average pooling on the dimensions corresponding to each label class in the fifth eigenvector, not only can the common features of each label class be obtained, but the dimensionality of these common features can also be reduced, which helps reduce the amount of subsequent computation.

[0019] In one implementation, the task feature extraction module is specifically used to perform vector multiplication and addition of the class scale factor vector of the support set and the fifth eigenvector to obtain the first eigenvector; the task feature extraction module is also used to perform vector multiplication and addition of the class scale factor vector and the sixth eigenvector of the query set to obtain the second eigenvector, where the sixth eigenvector is obtained by performing an embedding operation on the features of the query set; wherein the class scale factor vector is obtained by globally pooling the common feature vectors corresponding to the common features of each label class in the support set, and then inputting them into a fully connected neural network. In this way, the embodiment of the present application extracts a class scale factor vector for characterizing the common features of each label class from the support set input of the current task for the current sample labeling task, and then uses this class scale factor vector to perform feature enhancement on the support set feature vector and the query set feature vector, so that the difference features between the label classes of the support set and the query set in a single task are amplified, while the similarity features are relatively attenuated, so that the perception of the support set and the query set relative to the current task is enhanced, which is conducive to improving the labeling accuracy. For example, in the task of labeling "football" versus "basketball", the color and texture features will be enhanced, and in the task of "basketball" versus "flower", the shape features will be enhanced.

[0020] In one implementation, the task feature extraction module is specifically configured to use a normalized exponential function to map each dimension value after multiplying the class scale factor vector by the fifth eigenvector to a numerical range of 0 to 1, and then add the value to the fifth eigenvector to obtain the first eigenvector. In this way, the common features in the first eigenvector are compressed to a numerical range of 0 to 1 without affecting the original expression of the fifth eigenvector.

[0021] In one implementation, the task feature extraction module is specifically configured to use a normalized exponential function to map each dimension value after multiplying the class scale factor vector by the sixth eigenvector to a numerical range of 0 to 1, and then add the value to the sixth eigenvector to obtain the second eigenvector. In this way, in the second eigenvector, the common features are compressed to a numerical range of 0 to 1 without affecting the original expression of the sixth eigenvector.

[0022] In one implementation, the dynamic feature extraction module is specifically used to input the fifth feature vector into a convolutional neural network for feature extraction to obtain a task feature vector that characterizes the overall characteristics of the labeling task; the dynamic feature extraction module is also used to generate a random vector that obeys a multivariate Gaussian distribution, the random vector uses half of the dimension value of the task feature vector as the mean and the other half of the dimension value as the variance; the dynamic feature extraction module is also used to input the random vector into a fully connected neural network for dimensionality reduction processing to obtain the convolution kernel parameters. It can be seen that the convolution kernel parameters can be changed dynamically as the sample data of the support set changes. Therefore, the third and fourth feature vectors extracted using such convolution kernel parameters change dynamically as the sample data of the support set and the sample set change, which is conducive to improving labeling accuracy.

[0023] In one implementation, the sample labeling device further includes: a mutual attention module; the mutual attention module is used to obtain the seventh eigenvector of the support set and the eighth eigenvector of the query set, the seventh eigenvector is obtained by concatenating the first eigenvector and the third eigenvector, and the eighth eigenvector is obtained by concatenating the second eigenvector and the fourth eigenvector; the mutual attention module is also used to perform average pooling on the seventh eigenvector to obtain the ninth eigenvector of the support set; the mutual attention module is also used to obtain the local correlation eigenvector between the support set and the query set, the local correlation eigenvector is a correlation between the first eigenvector and the third eigenvector, and the eighth eigenvector is a correlation between the second eigenvector and the fourth eigenvector. The nine eigenvectors and the eighth eigenvector are converted in dimension and then matrix multiplied; the mutual attention module is also used to obtain the tenth eigenvector of the support set and the eleventh eigenvector of the query set, the tenth eigenvector is obtained by using the attention mechanism of the ninth eigenvector and the local correlation eigenvector, and the eleventh eigenvector is obtained by using the attention mechanism of the eighth eigenvector and the local correlation eigenvector; the classification module is used to determine the similarity of the query set relative to each label class in the support set based on the ninth eigenvector and the tenth eigenvector, and label the query set according to the similarity. The current common method generally adopts a convolutional neural network model when generating the attention weight distribution, thereby introducing additional parameters and increasing the difficulty of training. However, the embodiment of the present application does not introduce additional parameters when generating the attention weight distribution, which reduces the difficulty of training. In addition, since the embodiment of the present application does not introduce additional parameters, it can be directly transplanted to any trained small sample annotation framework without retraining.

[0024] In one implementation, the mutual attention module is specifically used to multiply the ninth eigenvector by the first attention weight distribution vector, and then accumulate the ninth eigenvector to obtain the tenth eigenvector. The first attention weight distribution vector is obtained by performing horizontal average pooling, normalized exponential function operation and dimensional conversion on the local correlation eigenvector.

[0025] In one implementation, the mutual attention module is specifically used to multiply the eighth eigenvector by the second attention weight distribution vector, and then accumulate the eighth eigenvector to obtain the eleventh eigenvector. The second attention weight distribution vector is obtained by performing vertical average pooling, normalized exponential function operation and dimensional conversion on the local correlation eigenvector.

[0026] In the embodiment of the present application, the local correlation feature vectors are subjected to simple average pooling and softmax operations to obtain respective attention weight distribution vectors, so that the support set and the query set can focus on similar feature parts during subsequent data comparison, thereby helping to improve the labeling accuracy.

[0027] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned aspects and methods of each implementation thereof.

[0028] In a fourth aspect, an embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the above-mentioned aspects and methods of their respective implementations.

[0029] In a fifth aspect, an embodiment of the present application further provides a chip system, which includes a processor for supporting the above-mentioned device or system to implement the functions involved in the above-mentioned aspects, for example, generating or processing the information involved in the above-mentioned method. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Schematic diagram of a sample of a meta-task training mode with small sample annotations shown in an embodiment of the present application;

[0031] Figure 2 The following is a schematic diagram showing the structure of a small sample annotation model based on meta-task;

[0032] Figure 3 This is an architectural diagram of a data annotation system exemplified in this application;

[0033] Figure 4 A schematic diagram illustrating the structure of a server for implementing the sample labeling method according to an embodiment of the present application is shown;

[0034] Figure 5 This is a flow chart of the sample labeling method provided in an embodiment of the present application;

[0035] Figure 6 This is a flow chart of step S102 of a sample labeling method provided in an embodiment of the present application;

[0036] Figure 7 2 is a schematic diagram of performing sample-level average pooling in step S201 according to an embodiment of the present application;

[0037] Figure 8 2 is a schematic diagram of performing channel-level global average pooling in step S202 according to an embodiment of the present application;

[0038] Figure 9 This is a flow chart of step S103 of a sample labeling method provided in an embodiment of the present application;

[0039] Figure 10 4 is a flowchart of steps S402 to S404 of the sample labeling method provided in an embodiment of the present application;

[0040] Figure 11 This is a flow chart of step S106 of a sample labeling method provided in an embodiment of the present application;

[0041] Figure 12 This is a schematic diagram of a sample labeling device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] Before explaining the technical solutions of the embodiments of the present application, the meanings of the technical terms in the technical field to which the embodiments of the present application belong are first listed and explained:

[0043] Meta-task: A minimal training / testing unit of a small sample classification task in a training / testing mode. In the embodiments of this application, a meta-task may correspond to the current labeling task in the claims.

[0044] Support set: A dataset with labels in the meta-task (i.e., known categories). The samples in the support set can be called support samples.

[0045] Query set: The dataset to be labeled in the meta-task (i.e., unknown category). The samples in the query set can be called query samples.

[0046] Label class: In data labeling tasks, the label class refers to the category of the sample. Samples of known categories can carry samples with labels.

[0047] In the following contents of the embodiments of the present application: the support set enhanced feature vector may correspond to the first feature vector in the claims; the query set enhanced feature vector may correspond to the second feature vector in the claims; the support set dynamic feature vector may correspond to the third feature vector in the claims; the support set feature vector may correspond to the fifth feature vector in the claims; the query set feature vector may correspond to the sixth feature vector in the claims; the query set dynamic feature vector may correspond to the fourth feature vector in the claims; the support set high-level dynamic feature vector may correspond to the seventh feature vector in the claims; the query set high-level dynamic feature vector may correspond to the eighth feature vector in the claims; the support set label class feature vector may correspond to the ninth feature vector in the claims; the support set label class interactivity feature vector may correspond to the tenth feature vector in the claims; the query set interactivity feature vector may correspond to the eleventh feature vector in the claims.

[0048] With the development of artificial intelligence (AI) technology, advanced machine learning methods, represented by deep learning, have been widely applied in various fields, including transportation, healthcare, and finance. Deep learning technology automatically learns representations of various types of data at different levels by building deep neural network models, further enabling applications such as data classification, data matching, and data recognition. Compared to traditional manual data feature mining, deep learning technology has superior data feature mining performance, resulting in outstanding performance in key applications in these fields.

[0049] However, due to their high complexity and large number of parameters, deep neural network models often require massive amounts of labeled data for model training, which carries a significant cost. Data labeling can be achieved using labeling models, which also have certain requirements for the size of sample data. Taking image data as an example, when the sample size of images for a certain label (category) is limited, the data labeling model training is prone to overfitting on that label, resulting in poor labeling results. Taking text data as an example, when the sample size of text for a certain label is limited, the text sample cannot reflect the context corresponding to the label, and the data labeling model cannot learn the contextual information corresponding to the label, resulting in poor labeling results. Taking speech data as an example, when the sample size of speech for a certain label is limited, these speech sounds may include blank spaces or noise, resulting in poor labeling results. Considering the time and labeling costs, as well as the long-tail effect, for specific image labeling tasks, only a small number of image samples can be provided for a given label.

[0050] Furthermore, the dataset to be labeled may change over time, and even samples with the same label may experience shifts in distribution. For example, a person's appearance may change with age, and the images of that person in a mobile phone gallery will naturally differ. These shifts in dataset and sample distribution can make existing labeling models less adaptable and degrade labeling results.

[0051] Therefore, how to use a small amount of labeled data to train a labeling model with excellent classification accuracy and the ability to adapt to dynamic data changes, that is, small sample labeling or small sample classification, is one of the important research directions in the industry. At present, the training and testing process of small sample classification can adopt the training mode of meta-task. Among them, the meta-task in the embodiment of this application refers to a minimum training / testing unit of the small sample classification task in the training / testing mode.

[0052] Figure 1 This is a sample diagram of a small sample labeled meta-task training mode shown in an embodiment of the present application. Figure 1 As shown in Figure 2, the meta-task training model can include a meta-training phase and a meta-testing phase, each of which contains multiple meta-tasks. The data used by each meta-task includes a support set and a query set. The support set refers to the dataset with labels in the meta-task (i.e., known categories), and the query set refers to the dataset to be labeled in the meta-task (i.e., unknown categories). Samples in the support set can be called support samples, and samples in the query set can be called query samples.

[0053] The meta-task can be expressed as N-way-K-shot, which means that the support set includes N categories, each category contains K samples, and the samples in the query set belong to one of the N categories. For example, in Figure 1 In the meta-task training mode shown, the meta-task is 5-way-1-shot, which means that the support set includes 5 categories, each category contains 1 image, and each image in the query set belongs to one of the 5 categories.

[0054] Currently, based on the meta-task training model, during the meta-training phase of a labeling model, labeled data (such as ImageNet data) is typically used to construct multiple meta-tasks, training a model with good generalization performance. During the meta-testing phase, a small amount of labeled data (with different categories than the data used in the training phase) is used as the support set, and the data to be labeled is used as the query set. These are fed into the trained labeling model, labeling the data to be labeled, completing the labeling process.

[0055] Figure 2 The following is a schematic diagram showing the structure of a small sample annotation model based on meta-task. Figure 2As shown in the figure, the annotation model includes: a feature extraction module (embedding module), which is used to simultaneously extract sample features of the support set and the query set through a multi-layer convolutional neural network (CNN); an image comparison module (image-to-class module), which is used to calculate the similarity of each query sample to a certain category (image-to-class) by using local features at the channel level of the image; and a classifier (classifier), which is used to perform classification prediction through the softmax layer of the similarity result through the softmax (function) operation.

[0056] Feature extraction is a concept in computer vision and image processing, which refers to using computers to extract image information and convert it into a digital vector representation. The softmax function, also known as the normalized exponential function, can "compress" a K-dimensional vector containing any real number into another K-dimensional real vector, so that each element ranges between (0, 1) and the sum of all elements is 1. Assuming that the input of the softmax layer is vector x, the calculation formula for the softmax operation on x is:

[0057]

[0058] However, in practical applications, the above-mentioned small sample annotation model has problems such as unreasonable feature extraction and poor adaptability to data changes, which affect the accuracy of data annotation. Specifically,

[0059] 1. The feature extraction module of this annotation model primarily extracts local features from images, without considering the overall requirements for feature extraction from different label settings in different annotation tasks. For example, in a labeling task involving both "football" and "basketball" labels, color and texture features are more important than shape features. However, in a labeling task involving both "basketball" and "flower," shape features may become crucial. This flaw in feature extraction becomes particularly prominent when there are many different labels.

[0060] 2. This annotation model treats images in the support and query sets independently, without considering the feature relationship between the support and query sets. As shown in Figure 2, the background color of the ladybugs in the query and support set images is different, but the ladybugs are similar. Therefore, when comparing similarities, the ladybug features should be emphasized.

[0061] 3. The parameters of the feature extraction module of the labeling model are fixed after training and cannot be adjusted dynamically. When new labels appear or the data of the original labels change dynamically, the model needs to be retrained, otherwise it will cause the model to fail.

[0062] In order to solve the above technical problems, an embodiment of the present application provides a sample labeling method.

[0063] The sample annotation method provided in the embodiment of the present application can be applied to a data annotation system. Figure 3 This is an architectural diagram of the data annotation system exemplified in this application. Figure 3 As shown, the data annotation system may include: a database and a data annotation device. The database stores a small number of labeled sample sets (i.e., support sets) and sample sets to be annotated (i.e., query sets). The data annotation device constructs a corresponding annotation model. The data annotation device uses the support set and sample set in the database as input to train the annotation model, thereby annotating the query set. The annotation results generated by the annotation model can be stored in the database or applied to subsequent scenarios such as data search and data recommendation.

[0064] The sample labeling method provided in the embodiments of the present application can be implemented by deploying a neural network model and computer program code in software form in a hardware computing environment. Available hardware computing environments include: personal computers, servers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, programmable consumer electronic devices, cloud servers, server instances, supercomputers, distributed computing environments including any of the above systems or devices, etc. Figure 4 A schematic diagram of the structure of a server for implementing the sample annotation method of an embodiment of the present application is shown as follows: Figure 4 As shown, the server may include a memory and a processor. The memory stores a database including a support set and a query set, a neural network model (annotation model) and a computer program code for implementing the sample annotation method of the embodiment of the present application, and the memory can also be used to store the annotation results. The processor may, for example, include at least one central processing unit (CPU), and / or at least one graphics processing unit (GPU), and / or at least one neural network processing unit (NPU), etc. The processor is used to run the neural network model and computer program code in the memory to implement the various steps of the sample annotation method of the embodiment of the present application and generate corresponding data.

[0065] In one implementation, the support set and query set data may also be stored on another device or in the cloud. When the method of the embodiment of the present application needs to be executed on a server, the server may obtain the support set and query set data from the other device or the cloud where the support set and query set are stored. The support set and query set data may be transmitted from the other device or the cloud to the server once or multiple times, which is not limited in the embodiment of the present application.

[0066] It should be noted that the method provided in the embodiments of the present application can be applied to various types of sample annotation scenarios, such as image annotation, named entity annotation, text annotation, voiceprint annotation, etc., and the embodiments of the present application are not limited to this. Next, the various embodiments of the sample annotation method of the embodiments of the present application are described exemplarily using image annotation as an example.

[0067] Example 1

[0068] Embodiment 1 of the present application provides an overall description of the technical solution of the sample labeling method. Figure 5 This is a flow chart of the sample labeling method provided by the embodiment of this application. Figure 5 As shown, the method may include the following steps S101 to S106:

[0069] Step S101 : performing low-level feature extraction on the support set and the query set to obtain a support set feature vector and a query set feature vector.

[0070] The purpose of low-level feature extraction is to convert samples in the support set and query set (such as images, named entities, text, voiceprints, etc.) from their original form, which is difficult for machines to understand, into a vector form that is easy for machines to understand and calculate. This process can also be called the embedding process. Different forms of samples require different low-level feature extraction methods. For example, for image samples, low-level sample features can be extracted through convolutional neural networks. For text samples such as named entities and text, low-level sample features can be extracted through word2vec. Common models involved in word2vec include Skip-gram and CBOW. The neural network models involved can include long short-term memory neural networks (LSTM) and recurrent neural networks (RNN). For audio information such as voiceprints, mel-frequency cepstral coefficients (MFCC) can be used to extract low-level sample features.

[0071] Taking a picture as an example, in the embodiment of the present application, the low-level sample feature corresponding to each picture can be a feature vector with a spatial dimension of c×h×w. Among them, c represents the number of channels of the sample feature, which is related to the number of convolution kernels used by the convolutional neural network. Generally speaking, each time a convolution kernel is used to convolve the picture, a channel vector can be obtained. The number of convolution kernels can be determined by those skilled in the art. The embodiment of the present application does not make specific restrictions. If those skilled in the art want to obtain a higher-dimensional vector, more convolution kernels can be used, and vice versa. h and w represent the height and width of the vector respectively, which can be related to the size of the convolution kernel and the step size of the convolution. The larger the convolution kernel and the larger the step size, the smaller h and w, and the smaller the convolution kernel and the smaller the step size, the larger h and w.

[0072] In addition, it should be noted that since the image is a pixel feature, it is not convenient for the convolutional neural network to perform convolution operations directly. Therefore, before inputting the image into the convolutional neural network, the histogram of oriented gradient (HOG) features of the image can be extracted first, and the image can be converted into a feature vector representing the color gradient, and then the feature vector representing the color gradient is input into the convolutional neural network.

[0073] In this embodiment of the present application, the support set feature vector may include the feature vectors of all labeled images in the support set, and the query set feature vector may include the feature vectors of all images to be labeled in the query set. For example:

[0074] The support set eigenvector can be a eigenvector f in a space with degree N×K×c×h×w s , here written as:

[0075] f s ∈R N×K×c×h×w

[0076] Where N represents the number of categories in the support set, and K represents the number of images in each category.

[0077] The query set feature vector can be a feature vector f with a spatial degree of N×?×c×h×w q , here written as:

[0078] f q ∈R N×？×c×h×w

[0079] Here, “?” represents the number of images to be labeled in the query set. For example, when the query set contains 1 image, the value of “?” is 1; when the query set contains 2 images, the value of “?” is 2, and so on.

[0080] It should be noted that the method of vectorizing the images in the support set and the query set provided in step S101 of the embodiment itself is only an exemplary method and does not constitute a specific limitation on the embodiments of the present application. Those skilled in the art can also use other methods in the field of machine vision or natural language processing to vectorize the images, which do not exceed the scope of protection of the embodiments of the present application.

[0081] Step S102: extract class common features from the support set feature vector, where the class common feature vector represents the common features of each label class in the support set. Apply the class common feature vector to the support set feature vector and the query set feature vector to obtain the support set enhanced feature vector and the query set enhanced feature vector.

[0082] Figure 6 This is a flowchart of step S102 of a sample labeling method provided in an embodiment of the present application.

[0083] like Figure 6 As shown, step S102 may include the following steps S201 to S2042:

[0084] Step S201 : Perform sample-level average pooling (AvgPool) on the dimension corresponding to each label class in the support set feature vector to obtain a class common feature vector.

[0085] Among them, average pooling is a feature processing method that takes the average value of multiple vectors. It can retain the significant features of the vector, reduce the feature dimension, and increase the perceptual field of view of the convolution kernel when it is used subsequently.

[0086] Figure 7 2 is a schematic diagram of performing sample-level average pooling in step S201 of the method according to an embodiment of the present application.

[0087] like Figure 7 As shown in , sample-level average pooling refers to treating the dimension c×h×w corresponding to each sample (i.e., each image) in the support set feature vector as a vector. In this way, when each label class contains K samples, the label class contains K vectors of dimension c×h×w. Sample-level average pooling is to calculate the average value of the values ​​at the same position in these K c×h×w vectors, so that for each label class, a common feature vector of dimension c×h×w that represents this label class can be obtained. Therefore, for a support set feature vector containing N label classes, its corresponding class common feature vector F s It can be expressed as:

[0088]

[0089] It should be noted here that the use of average pooling to obtain the class common feature vector is only an example in step S201 of the embodiment of the present application. This means that those skilled in the art can also use other pooling methods to obtain the class common feature vector, such as maximum pooling MaxPooling, etc., which do not exceed the protection scope of the embodiment of the present application.

[0090] In step S202 , global average pooling (GAP) at the channel level is performed on the class common feature vector to obtain a class channel feature vector.

[0091] Among them, the class channel feature vector includes the channel common features of each label class in the support set.

[0092] Global average pooling refers to a special processing method that averages all dimension values ​​in a vector.

[0093] Figure 8 2 is a schematic diagram of performing channel-level global average pooling in step S202 of the method of an embodiment of the present application.

[0094] like Figure 8 As shown, the global average pooling at the channel level refers to the class common feature vector F s ∈R N×c×h×w The vector with the dimension of c×h×w representing the common features of each label class is used as a vector, and all dimensions of the height h and width w in this vector are averaged. After global average pooling, the dimension of each vector with the dimension of c×h×w becomes c. Therefore, for the class common feature vector F containing N label classes, s ∈R N×c×h×w For example, its corresponding channel-like feature vector F′ s It can be expressed as:

[0095]

[0096] It should be noted here that the use of global average pooling to obtain the channel-like feature vector is only an example in step S202 of the embodiment of the present application. This means that those skilled in the art can also use other pooling methods to obtain the channel-like feature vector, such as global maximum pooling, etc., which do not exceed the protection scope of the embodiment of the present application.

[0097] In step S203, the class channel feature vector is input into a fully connected neural network (FC) to obtain a class scale factor vector of the support set.

[0098] The fully connected neural network is used to transform the class channel feature vector F′ s ∈R N×c From a flat vector dimensionality reduction to an R N In the one-dimensional space, it is equivalent to the channel feature vector F′ s ∈R N×c The features at different positions in the same label class are integrated and output, and the common features of each label class are represented by a factor value. For the class channel feature vector F′ containing N label classes s ∈R N×c For example, the corresponding class scale factor vector α can be expressed as:

[0099] α=g θ (F′ s )∈R N×1

[0100] Among them, g θ represents a fully connected neural network.

[0101] In one implementation, the fully connected neural network may be a two-layer fully connected neural network.

[0102] Step S2041 , using a normalized exponential function to map each dimension value after multiplying the class scale factor vector and the support set eigenvector to a numerical range of 0 to 1, and then adding it to the support set eigenvector to obtain the support set enhanced eigenvector.

[0103] Among them, the normalized exponential function, such as the softmax function, is used to "compress" a multidimensional vector containing arbitrary real numbers into another multidimensional real vector, so that the range of each element is between (0, 1) and the sum of all elements is 1.

[0104] For example, assuming that the softmax function input is a vector x, the formula for calculating the softmax function on x is:

[0105]

[0106] Among them, x i Represents the value of the i-th dimension in vector x.

[0107] Then, step S2041 obtains the support set enhanced feature vector f′ s The method can be expressed by the following formula:

[0108] f′ s =softmax(α*f s )+f s ∈R N×K×c×h×w

[0109] In step S2042 , a normalized exponential function is used to map each dimension value after multiplying the class scale factor vector and the query set feature vector to a numerical range of 0 to 1, and then added to the query set feature vector to obtain a query set enhanced feature vector.

[0110] Based on the content of the softmax function output above, step S2042 obtains the query set enhanced feature vector f′ q The method can be expressed by the following formula:

[0111] f′ q =softmax(α*f q )+f q ∈R N×？×c×h×w

[0112] In step S102 of the embodiment of the present application, for the current sample labeling task, a class scale factor vector for characterizing the common features of each label class is extracted from the support set input for the current task, and then the class scale factor vector is used to perform feature enhancement on the support set feature vector and the query set feature vector, so that the difference features between the label classes of the support set and the query set in a single task are amplified, while the similarity features are relatively attenuated, so that the perception of the support set and the query set relative to the current task is enhanced, which is conducive to improving the labeling accuracy. For example, in the task of labeling "football" to "basketball", the color and texture features will be enhanced, and in the task of labeling "basketball" to "flower", the shape features will be enhanced.

[0113] Step S103: Use a dynamic parameter generator to generate convolution kernel parameters from the support set feature vector, and use a convolutional neural network loaded with the convolution kernel parameters to perform feature extraction on the support set feature vector and the query set feature vector respectively to obtain the support set dynamic feature vector and the query set dynamic feature vector.

[0114] Among them, the dynamic parameter generator can be composed of a convolutional neural network and a fully connected neural network.

[0115] Figure 9 This is a flowchart of step S103 of a sample labeling method provided in an embodiment of the present application.

[0116] like Figure 9 As shown, based on the structure of the above dynamic parameter generator, step S103 may include the following steps S301 to S304:

[0117] In step S301, the support set feature vector is input into a convolutional neural network for feature extraction to obtain a task feature vector representing the overall features of the current labeling task.

[0118] The task feature vector can be a vector f with a dimension of 2m t , here written as:

[0119] f t ∈R 2m

[0120] Among them, the numerical value of m can be determined by the size of the convolution kernel and the convolution step size adopted by the convolutional neural network in step S301. The larger the size of the convolution kernel and the convolution step size, the smaller the numerical value of m, which means that the overall features of the current labeling task extracted are rougher. The fewer the dimensions of the convolution kernel, the larger the numerical value of m, which means that the overall features of the current labeling task extracted are finer. Therefore, those skilled in the art can set the numerical value of m according to the desired feature accuracy, and the embodiments of the present application are not limited to this.

[0121] Generally speaking, since feature extraction of the support set feature vector by a convolutional neural network is usually a dimensionality reduction operation, the size of 2m is usually smaller than the original dimension of the support set feature vector. For example, 2m is usually smaller than N×K×c×h×w.

[0122] In step S302 , a random vector obeying a multivariate Gaussian distribution is randomly generated by taking half of the dimension values ​​of the task feature vector as the mean and the other half of the dimension values ​​as the variance.

[0123] In one implementation, the first half of the task feature vector’s dimension value can be used as the mean μ and the second half of the dimension value can be used as the variance σ to generate a random vector r, where the dimension of the random vector r can also be 2m or other dimensions. For example, when the task feature vector f t ∈R 2m When the first half dimension value can be expressed as f t [:m], the second half dimension value can be expressed as f t [m:], then the multivariate Gaussian distribution N obtained can be expressed as N(μ=f t [:m],σ=f t [m:]), the random vector r that obeys the multivariate Gaussian distribution N can be expressed as:

[0124] r∈R 2m

[0125] In another implementation, the random vector r can be generated by taking the second half dimension value of the task feature vector as the mean μ and the first half dimension value as the variance σ. For example, when the task feature vector f t ∈R 2m When the multivariate Gaussian distribution N can be expressed as N(μ=f t [m:],σ=f t[:m]), and then generate a random vector r that obeys the multivariate Gaussian distribution N.

[0126] In some other implementations, the mean μ and variance σ of the multivariate Gaussian distribution N can also be determined from the task feature vector by other methods, for example: extracting the dimension value as the mean μ and the dimension value as the variance from the task feature vector in an interval manner, etc. The embodiments of the present application are not limited to this.

[0127] Step S303: Input the random vector into a fully connected neural network for dimensionality reduction to obtain convolution kernel parameters.

[0128] A fully connected neural network is used to transform a random vector r∈R 2m Perform dimensionality reduction to obtain the convolution kernel parameters with lower dimensions:

[0129] kernel=g θ (r)∈R n

[0130] Among them, g θ Represents a fully connected neural network. The dimension n of the kernel is usually less than 2m. It can be specifically determined according to the parameters of the input layer and output layer of the fully connected neural network. The embodiment of the present application does not limit this.

[0131] Step S304 : Using a convolutional neural network loaded with the convolution kernel parameters, feature extraction is performed on the support set feature vector and the query set feature vector respectively to obtain a support set dynamic feature vector and a query set dynamic feature vector.

[0132] In a specific implementation, the convolution kernel parameters obtained in step S303 can be used to construct two convolutional neural networks, and then the support set feature vector is input into one of the convolutional neural networks for feature extraction to obtain a support set dynamic feature vector, and the query set feature vector is input into the other convolutional neural network for feature extraction to obtain a query set dynamic feature vector.

[0133] In step S103 of the embodiment of the present application, a dynamic parameter generator is introduced. During the training process of the annotation model, the parameters of the convolutional neural network and the fully connected layer can be trained. In the test and production environments, the trained dynamic parameter generator can be used to generate convolution kernel parameters from the support set feature vector. It can be seen that the convolution kernel parameters can be dynamically changed as the sample data of the support set changes. Therefore, the support set dynamic feature vector and the query set dynamic feature vector extracted using such convolution kernel parameters change dynamically as the sample data of the support set and the sample set change, which is conducive to improving the annotation accuracy.

[0134] Step S104 , concatenating the support set enhanced feature vector and the support set dynamic feature vector to obtain a support set high-level dynamic feature vector, and concatenating the query set enhanced feature vector and the query set dynamic feature vector to obtain a query set high-level dynamic feature vector.

[0135] In specific implementation, such as Figure 9 As shown, the support set enhanced feature vector and the support set dynamic feature vector can be spliced ​​in the channel dimension to obtain the support set high-level dynamic feature vector. In this way, the support set high-level dynamic feature vector contains both the original c channels and the channels corresponding to the support set dynamic feature vector. The number of channels can be recorded as c′. Then, the support set high-level dynamic feature vector f″ s It can be expressed as:

[0136] f″ s =∈R N×K×c′×h×w

[0137] Similarly, the query set enhanced feature vector and the query set dynamic feature vector can be concatenated in the channel dimension to obtain the query set high-level dynamic feature vector. In this way, the query set high-level dynamic feature vector contains both the original c channels and the channels corresponding to the query set dynamic feature vector. The number of channels can be recorded as c′. Then, the query set high-level dynamic feature vector Q can be expressed as:

[0138] Q=∈R N×？×c′×h×w

[0139] Step S105, perform average pooling on the support set high-level dynamic feature vector to obtain the support set label class feature vector, obtain the local correlation feature vector of the support set label class feature vector and the query set high-level dynamic feature vector, apply the local correlation feature vector to the support set label class feature vector and the query set high-level dynamic feature vector respectively, and obtain the support set label class interactivity feature vector and the query set interactivity feature vector.

[0140] Based on the structure of the above dynamic parameter generator, step S105 may include the following steps S401 to S404:

[0141] Step S401 : performing sample-level average pooling on the support set high-level dynamic feature vector to obtain the support set label feature vector.

[0142] Sample-level average pooling refers to treating the dimension c'×h×w corresponding to each sample (i.e., each image) in the support set high-level dynamic feature vector as a vector. In this way, when each category contains K samples, the category contains K vectors of dimension c'×h×w. Sample-level average pooling is to calculate the average value of the values ​​at the same position in these K c'×h×w vectors, so that for each category, a vector of dimension c'×h×w that represents the common characteristics of this category can be obtained. Therefore, for a support set high-level dynamic feature vector containing N categories, its corresponding support set label class feature vector P s It can be expressed as:

[0143]

[0144] It should be noted here that the use of average pooling to obtain the support set label class feature vector is only an example in step S401 of the embodiment of the present application. This means that those skilled in the art can also use other pooling methods to obtain the support set label class feature vector, such as maximum pooling MaxPooling, etc., which do not exceed the protection scope of the embodiments of the present application.

[0145] Step S402 : After dimension conversion of the support set label feature vector and the query set high-level dynamic feature vector, matrix multiplication is performed to obtain a local correlation feature vector.

[0146] Figure 10 It is a flowchart of steps S402 to S404 of the sample labeling method provided in an embodiment of the present application.

[0147] like Figure 10 As shown, for ease of description, the support set label class feature vector P and the query set high-level dynamic feature vector Q can be simplified as three-dimensional solid vectors of C×H×W. First, the support set label class feature vector P is converted from a three-dimensional solid vector of C×H×W to a two-dimensional plane vector with a width of C and a height of H×W. Similarly, the query set high-level dynamic feature vector Q can be converted from a three-dimensional solid vector of C×H×W to a two-dimensional plane vector with a width of C and a height of H×W, and then transposed to a two-dimensional plane vector with a width of H×W and a height of C. That is, the width of the two-dimensional plane vector corresponding to the support set label class feature vector P is equal to the height of the two-dimensional plane vector corresponding to the query set high-level dynamic feature vector Q, and the height of the two-dimensional plane vector corresponding to the support set label class feature vector P is equal to the width of the two-dimensional plane vector corresponding to the query set high-level dynamic feature vector Q.

[0148] Next, the two-dimensional plane vector with a width of C and a height of H×W obtained by converting the support set label class feature vector P is matrix multiplied with the two-dimensional plane vector with a width of C and a height of H×W obtained by converting the query set high-level dynamic feature vector Q to obtain a two-dimensional plane vector with a width of H×W and a height of H×W, namely the local correlation feature vector M.

[0149] In step S403, the local correlation feature vector is subjected to horizontal average pooling, normalized exponential function operation and dimensional conversion to obtain a first attention weight distribution vector, and the local correlation feature vector is subjected to vertical average pooling, normalized exponential function operation and dimensional conversion to obtain a second attention weight distribution vector.

[0150] like Figure 10 As shown:

[0151] Average pooling in the horizontal direction refers to taking the average of the dimension values ​​in the same row of the vector. Therefore, after the local correlation feature vector M with a width of H×W and a height of H×W is averaged in the horizontal direction, a vector with a width of 1 and a height of H×W can be obtained. Then, the obtained vector is subjected to the normalized exponential function softmax operation so that the values ​​of each dimension are normalized to the range of 0 to 1. Finally, the vector after the softmax operation is dimensionally transformed to obtain a vector with a width of W and a height of H, which is called the first attention weight distribution vector A. p .

[0152] Vertical average pooling refers to taking the average of the dimension values ​​in the same column of the vector. Therefore, after the local correlation feature vector M with a width of H×W and a height of H×W is averaged in the vertical direction, a vector with a width of H×W and a height of 1 can be obtained. Then, the obtained vector is subjected to the normalized exponential function softmax operation so that the values ​​of each dimension are normalized to the range of 0 to 1. Finally, the vector after the softmax operation is dimensionally transformed to obtain a vector with a width of W and a height of H, which is called the second attention weight distribution vector A. q .

[0153] It should be noted here that the use of average pooling to process the local correlation feature vector M is only an example in step S403 of the embodiment of the present application, which means that those skilled in the art can also use other pooling methods, such as maximum pooling MaxPooling, etc., which do not exceed the scope of protection of the embodiment of the present application.

[0154] In step S404, the support set label class feature vector is multiplied by the first attention weight distribution vector, and the support set label class feature vector is accumulated to obtain the support set label class interactivity feature vector, and the query set high-level dynamic feature vector is multiplied by the second attention weight distribution vector, and the query set high-level dynamic feature vector is accumulated to obtain the query set interactivity feature vector.

[0155] Combine Figure 10 The method of obtaining the support set label class interactivity feature vector P′ in step S404 can be expressed by the following formula:

[0156] P′=P*M+P∈R C×H×W

[0157] The method for obtaining the query set interactivity feature vector Q′ in step S404 can be expressed by the following formula:

[0158] Q′=Q*M+Q∈R C×H×W

[0159] In step S105, the embodiment of the present application performs a simple average pooling and softmax operation on the local correlation feature vector M of the support set label class feature vector and the query set high-level dynamic feature vector Q to obtain respective attention weight distribution vectors, so that the support set and the query set can focus on similar feature parts when performing subsequent data comparison, which is conducive to improving the annotation accuracy. In addition, the current common method generally adopts a convolutional neural network model when generating the attention weight distribution, thereby introducing additional parameters and increasing the training difficulty. However, the embodiment of the present application does not introduce additional parameters when generating the attention weight distribution in step S105, thereby reducing the training difficulty. In addition, since the embodiment of the present application does not introduce additional parameters in step S105, it can be directly transplanted into any trained small sample annotation framework without the need for retraining.

[0160] Step S106 , calculating the similarity between the query set interactivity feature vector and each label class in the support set label class interactivity feature vector, and using the label class with the highest similarity to label the sample to be labeled.

[0161] Step S106 can be applied to the testing and production environment stages of the annotation model. Figure 11 As shown in Figure 1, channel-level local features can be used to calculate the similarity between the interactive feature vector of each query sample in the query set's interactive feature vector and the interactive feature vectors of each label class in the support set's label class interactive feature vector. This allows us to obtain the similarity between each query sample and a particular label class. Then, based on the similarity calculation results, the sample to be labeled is labeled with the label class with the highest similarity.

[0162] The similarity can also be referred to as the similarity distance, and the highest similarity corresponds to the shortest similarity distance. In the embodiment of the present application, the similarity can be a cosine distance, a Euclidean distance, etc., which is not specifically limited here.

[0163] In addition, during the training phase of the annotation model, such as Figure 11 As shown, the query set interactivity feature vector can also be input into a global classifier for global label prediction. The global classifier can, for example, consist of a fully connected layer and a softmax layer. The role of the global classifier is to assign the query set interactivity feature vector (i.e., the query set after attention) to the global label of the entire task, so that the labeling model not only focuses on the label of the current meta-task, but also on other labels in the entire training set. Therefore, the global classifier can reduce the risk of overfitting in the current meta-task training.

[0164] It can be seen from the above technical solutions that the sample labeling method provided by the embodiment of the present application first extracts a class scale factor vector for characterizing the common features of each label class from the support set input of the current task for the current sample labeling task, and then uses this class scale factor vector to perform feature enhancement on the support set feature vector and the query set feature vector, so that the difference features between the label classes of the support set and the query set in a single task are amplified, while the similarity features are relatively attenuated, so that the perception of the support set and the query set relative to the current task is enhanced, which is conducive to improving the sample labeling accuracy; then, a dynamic parameter generator is introduced, and the dynamic parameter generator can be trained during the training process of the labeling model, and can be used in test and production environments. The trained dynamic parameter generator generates convolution kernel parameters from the support set feature vector, so that the convolution kernel parameters can be dynamically changed as the sample data of the support set changes. Furthermore, the support set dynamic feature vector and the query set dynamic feature vector extracted by the convolution kernel parameters change dynamically as the sample data of the support set and the sample set change, which is beneficial to improving the sample labeling accuracy; then, a simple average pooling and softmax operation is performed on the local correlation feature vectors of the support set label class feature vector and the query set high-level dynamic feature vector to obtain their respective attention weight distribution vectors, so that the support set and the query set can focus on similar feature parts during subsequent data comparison, which is beneficial to improving the sample labeling accuracy.

[0165] In order to verify that the sample labeling method and labeling model provided in the embodiments of the present application can improve the accuracy of sample labeling, the applicant conducted some tests using the benchmark dataset miniImageNet in the field of small sample image classification.

[0166] MiniImageNet is a subset of ImageNet, the dataset used in the ILSVRC (Large Scale Visual Recognition Challenge). MiniImageNet contains 100 image categories, each containing 600 images. In testing, 64, 16, and 20 of these 100 image categories were divided into training, validation, and test sets, respectively, and the images were resized to 84×84 pixels.

[0167] Based on the N-way K-shot meta-task definition commonly used for small-sample image classification problems, N categories are randomly sampled in each meta-task. K images from each category are randomly selected to form the support set. The remaining images from that category are then selected into the query set. During the test phase, the query set contains a consistent number of images for each category, and the model's performance is evaluated through prediction accuracy.

[0168] The training process uses a multi-round scenario training approach. A total of 100 rounds of training are performed on the miniImageNet dataset, with 1200 N-way K-shot meta-tasks randomly sampled in each round for training. After every 10 rounds of training for the first 60 rounds, 600 meta-tasks are randomly sampled from the validation set to evaluate the model's current training performance and save the model parameters. After 60 rounds, the same model performance evaluation is performed after each training round.

[0169] During the model testing phase, 2000 meta-tasks are randomly sampled from the test set for testing. To account for potential model overfitting during training, the best performing model is selected for testing based on the interim evaluation results obtained during the training phase using the validation set data. After testing, the average classification accuracy across the 2000 meta-tasks and the corresponding 95% confidence interval are calculated.

[0170] Taking the 5-way 1-shot and 5-way 5-shot meta-tasks as an example, the final test results and comparison with other models are shown in the following table:

[0171] Table 1. 5-way meta-task test accuracy (95% confidence interval) of miniImageNet dataset

[0172]

[0173]

[0174] It can be seen that the sample labeling method provided by the embodiment of the present application has improved labeling accuracy compared with the sample labeling methods of various prior arts.

[0175] In addition, it should be noted that each method step of the sample labeling method provided in the embodiment of the present application can be independently transplanted and integrated into the existing sample labeling solution, which can also achieve the effect of improving the labeling accuracy.

[0176] For example, to verify the effect of step S102 of the present application in improving the annotation accuracy, the applicant compared the sample annotation method of the embodiment of the present application after removing step S102 with the complete method, and obtained the following test results:

[0177] Table 1 Test accuracy of the complete sample annotation method and the method removing step S102

[0178]

[0179] It can be seen that step S103 of the sample labeling method provided in the embodiment of the present application can improve the labeling accuracy.

[0180] The effects of other steps of the sample labeling method provided in the embodiment of the present application on improving accuracy can be tested by similar methods and will not be repeated here.

[0181] The above embodiments introduce the various steps of the sample annotation method provided by the present application. It is understandable that, in order to implement the above functions, devices such as servers include hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of the various examples described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0182] For example, devices such as servers can implement corresponding functions through software modules.

[0183] In one embodiment, Figure 12As shown, the device for implementing the above-mentioned sample labeling method includes: a feature enhancement module 120, which is used to extract class common features from the support set feature vector, the class common feature vector represents the common features of each label class in the support set, and the class common feature vector is applied to the support set feature vector and the query set feature vector respectively to obtain the support set enhanced feature vector and the query set enhanced feature vector; a dynamic feature extraction module 130, which is used to generate convolution kernel parameters from the support set feature vector using a dynamic parameter generator, the dynamic parameter extractor is composed of a convolutional neural network and a fully connected neural network, and the convolution neural network loaded with the convolution kernel parameters is used to extract features from the support set feature vector and the query set feature vector respectively to obtain the support set dynamic feature vector and the query set dynamic feature vector; a mutual attention module 140, which is used to combine the support set enhanced feature vector with the support set dynamic feature vector. Vector splicing is performed to obtain a support set high-level dynamic feature vector, and the query set enhanced feature vector is spliced ​​with the query set dynamic feature vector to obtain a query set high-level dynamic feature vector; the mutual attention module 140 is also used to perform average pooling on the support set high-level dynamic feature vector to obtain a support set label class feature vector, obtain a local correlation feature vector between the support set label class feature vector and the query set high-level dynamic feature vector, and apply the local correlation feature vector to the support set label class feature vector and the query set high-level dynamic feature vector respectively to obtain a support set label class interactivity feature vector and a query set interactivity feature vector; the classification module 150 is used to calculate the similarity between each label class in the query set interactivity feature vector and the support set label class interactivity feature vector, and use the label class with the highest similarity to label the samples to be labeled in the query set.

[0184] The technical solution provided in the embodiment of the present application simultaneously introduces the common features of each label class in the support set, the support set and query set and dynamic features that change based on the labeling task, and the mutual correlation features of the support set label class and the query set, and generates a support set label class interactive feature vector and a query set interactive feature vector containing the above features. Therefore, the support set label class interactive feature vector and the query set interactive feature vector have a more reasonable feature expression of the support set and query set, are more adaptable to data changes, and improve the sample labeling accuracy.

[0185] In one implementation, the feature enhancement module 120 specifically includes: a first pooling unit 121, which is used to perform sample-level average pooling on the dimension corresponding to each label class in the support set feature vector to obtain a class common feature vector; a second pooling unit 122, which is used to perform channel-level global average pooling on the class common feature vector to obtain a class channel feature vector; a class scale factor generation unit 123, which is used to input the class channel feature vector into a fully connected neural network to obtain a class scale factor vector of the support set; an enhanced feature generation unit 124, which is used to use a normalized exponential function to map each dimension value after multiplying the class scale factor vector by the support set feature vector to a numerical range of 0 to 1, and then add the value to the support set feature vector to obtain a support set enhanced feature vector, and, use a normalized exponential function to map each dimension value after multiplying the class scale factor vector by the query set feature vector to a numerical range of 0 to 1, and then add the value to the query set feature vector to obtain a query set enhanced feature vector. In this way, the embodiment of the present application extracts a class scale factor vector for characterizing the common features of each label class from the support set input of the current task for the current sample labeling task, and then uses this class scale factor vector to perform feature enhancement on the support set feature vector and the query set feature vector, so that the difference features between the label classes of the support set and the query set in a single task are amplified, while the similarity features are relatively attenuated, so that the perception of the support set and the query set relative to the current task is enhanced, which is conducive to improving the labeling accuracy. For example, in the task of labeling "football" versus "basketball", the color and texture features will be enhanced, and in the task of "basketball" versus "flower", the shape features will be enhanced.

[0186] In one implementation, the dynamic feature extraction module 130 specifically includes: a task feature extraction unit 131, which is used to input the support set feature vector into the convolutional neural network for feature extraction to obtain a task feature vector that represents the overall characteristics of the current labeling task; a random vector generation unit 132, which is used to randomly generate a random vector that obeys the multivariate Gaussian distribution with half of the dimension value of the task feature vector as the mean and the other half of the dimension value as the variance; a convolution kernel parameter generation unit 133, which is used to input the random vector into the fully connected neural network for dimensionality reduction processing to obtain the convolution kernel parameters; a dynamic feature extraction unit 134, which is used to use the convolution neural network loaded with the convolution kernel parameters to respectively perform dimensionality reduction on the support set feature vector. Vector and query set feature vector are used for feature extraction to obtain support set dynamic feature vector and query set dynamic feature vector. In this way, the embodiment of the present application introduces a dynamic parameter generator, and the convolutional neural network and fully connected layer parameters can be trained during the training process of the annotation model. In the test and production environment, the trained dynamic parameter generator can be used to generate convolution kernel parameters from the support set feature vector. It can be seen that the convolution kernel parameters can be dynamically changed as the sample data of the support set changes. Therefore, the support set dynamic feature vector and query set dynamic feature vector extracted using such convolution kernel parameters change dynamically as the sample data of the support set and the sample set change, which is conducive to improving the annotation accuracy.

[0187] In one implementation, the random vector generating unit 132 is specifically configured to randomly generate a random vector by using the first half dimension value of the task feature vector as the mean and the second half dimension value as the variance.

[0188] In one implementation, the mutual attention module 140 specifically includes: a first feature splicing unit 141, which is used to splice the support set enhanced feature vector and the support set dynamic feature vector in the channel dimension to obtain a support set high-level dynamic feature vector.

[0189] In one implementation, the mutual attention module 140 specifically includes: a second feature concatenation unit 142, configured to concatenate the query set enhanced feature vector and the query set dynamic feature vector in the channel dimension to obtain a query set high-level dynamic feature vector.

[0190] In one implementation, the mutual attention module 140 specifically includes: a third pooling unit 143, which is used to perform sample-level average pooling on the support set high-level dynamic feature vector to obtain a support set label class feature vector; a local correlation feature generation unit 144, which is used to perform matrix multiplication on the support set label class feature vector and the query set high-level dynamic feature vector after dimension conversion to obtain a local correlation feature vector; a weight distribution generation unit 145, which is used to perform horizontal average pooling, normalized exponential function operation and dimension conversion on the local correlation feature vector to obtain a first attention weight distribution vector, and to perform vertical average pooling, normalized exponential function operation and dimension conversion on the local correlation feature vector to obtain a second attention weight distribution vector; an interactivity feature generation unit 146, which is used to multiply the support set label class feature vector with the first attention weight distribution vector, and then accumulate the support set label class feature vector to obtain a support set label class interactivity feature vector, and to multiply the query set high-level dynamic feature vector with the second attention weight distribution vector, and then accumulate the query set high-level dynamic feature vector to obtain a query set interactivity feature vector. In this way, the embodiment of the present application performs a simple average pooling and softmax operation on the local correlation feature vectors of the support set label class feature vector and the query set high-level dynamic feature vector to obtain respective attention weight distribution vectors, so that the support set and the query set can focus on similar feature parts when performing subsequent data comparison, thereby helping to improve the annotation accuracy. In addition, the current common method generally adopts a convolutional neural network model when generating the attention weight distribution, thereby introducing additional parameters and increasing the training difficulty, while the embodiment of the present application does not introduce additional parameters when generating the attention weight distribution in step S105, thereby reducing the training difficulty. In addition, since the embodiment of the present application does not introduce additional parameters in step S105, it can be directly transplanted into any trained small sample annotation framework without the need for retraining.

[0191] In one implementation, the local correlation feature generation unit 144 is specifically used to: convert the support set label class feature vector and the query set high-level dynamic feature vector from three-dimensional stereo vectors into two-dimensional plane vectors, wherein the width of the two-dimensional plane vector corresponding to the support set label class feature vector is equal to the height of the two-dimensional plane vector corresponding to the query set high-level dynamic feature vector, and the height of the two-dimensional plane vector corresponding to the support set label class feature vector is equal to the width of the two-dimensional plane vector corresponding to the query set high-level dynamic feature vector; multiply the two-dimensional plane vector corresponding to the support set label class feature vector with the two-dimensional plane vector corresponding to the query set high-level dynamic feature vector to obtain the local correlation feature vector.

[0192] In one implementation, the device also includes: a low-level feature extraction module 110, which is used to perform low-level feature extraction on the support set and the query set to obtain a support set feature vector and a query set feature vector. The low-level feature extraction includes using a convolutional neural network to perform feature extraction on the support set and the query set.

[0193] For example, devices such as servers can implement corresponding functions through hardware modules.

[0194] In one embodiment, Figure 4 As shown, the device for implementing the above-mentioned sample labeling method includes: a processor 210 and a memory 220; wherein the memory 220 includes program instructions, and when the program instructions are executed by the processor 210, the server is used to perform the following steps: extracting class common features from the support set feature vector, the class common feature vector represents the common features of each label class in the support set, and applying the class common feature vector to the support set feature vector and the query set feature vector respectively to obtain the support set enhanced feature vector and the query set enhanced feature vector; using a dynamic parameter generator to generate convolution kernel parameters from the support set feature vector, the dynamic parameter generator is composed of a convolutional neural network and a fully connected neural network, and using the convolution neural network loaded with the convolution kernel parameters to extract features from the support set feature vector and the query set feature vector respectively to obtain the support set dynamic feature vector and the query set dynamic feature vector. feature vector; concatenate the support set enhanced feature vector with the support set dynamic feature vector to obtain the support set high-level dynamic feature vector, and concatenate the query set enhanced feature vector with the query set dynamic feature vector to obtain the query set high-level dynamic feature vector; perform average pooling on the support set high-level dynamic feature vector to obtain the support set label class feature vector, obtain the local correlation feature vector of the support set label class feature vector and the query set high-level dynamic feature vector, apply the local correlation feature vector to the support set label class feature vector and the query set high-level dynamic feature vector respectively, and obtain the support set label class interactivity feature vector and the query set interactivity feature vector; calculate the similarity between each label class in the query set interactivity feature vector and the support set label class interactivity feature vector, and use the label class with the highest similarity to label the samples to be labeled in the query set.

[0195] An embodiment of the present application further provides a computer storage medium, in which computer instructions are stored. When the computer storage medium is run on a computer, the computer is enabled to execute the above-mentioned methods.

[0196] The embodiment of the present application also provides a computer program product containing instructions, which, when executed on a computer, enables the computer to execute the above-mentioned methods.

[0197] The present application also provides a chip system. The chip system includes a processor for supporting the above-mentioned apparatus or device to implement the functions involved in the above aspects, such as generating or processing the information involved in the above-mentioned method. In one possible design, the chip system also includes a memory for storing the necessary program instructions and data for the above-mentioned apparatus or device. The chip system can be composed of a chip or can include a chip and other discrete devices.

[0198] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for labeling an image, characterized in that: include: Get the low-level features corresponding to each image in the support set and query set; Determining a scale factor corresponding to each label class based on the low-level features corresponding to each image in the support set and the label class corresponding to each image in the support set; Based on the scale factor corresponding to each of the label classes, feature enhancement is performed on the low-level features corresponding to each image in the support set and the query set to obtain enhanced features corresponding to each image in the support set and the query set; generating dynamic convolution kernel parameters that dynamically change with low-level features corresponding to each image in the support set; Based on the low-level features and enhanced features corresponding to each image in the support set and the query set, using a convolutional neural network including the dynamic convolution kernel parameters, obtaining high-level dynamic features corresponding to each image in the support set and the query set; Determining the similarity of each image in the query set to each of the label classes based on the high-level dynamic features corresponding to each image in the support set and the query set; Based on the similarity, each image in the query set is labeled.

2. The method according to claim 1, characterized in that The method of determining the scale factor corresponding to each label class based on the low-level features corresponding to each image in the support set and the label class corresponding to each image in the support set includes: based on the label class corresponding to each image in the support set, performing pooling processing on the low-level features corresponding to the images of each label class in the support set, and then obtaining the scale factor corresponding to each label class using a fully connected neural network.

3. The method according to claim 1, characterized in that Generating dynamic convolution kernel parameters that dynamically change with low-level features corresponding to each image in the support set includes: Extracting task features corresponding to the current labeling task from the low-level features corresponding to each image in the support set; The dynamic convolution kernel parameters are determined based on the task characteristics corresponding to the current labeling task.

4. The method according to any one of claims 1 to 3, characterized in that The step of enhancing the low-level features corresponding to each image in the support set and the query set based on the scale factor corresponding to each label class to obtain enhanced features corresponding to each image in the support set and the query set includes: Using a normalized exponential function, mapping each dimension value after multiplying the scale factor corresponding to each label class by the low-level feature corresponding to each image in the support set to a target numerical range, and then adding it to the low-level feature corresponding to each image in the support set to obtain the enhanced feature corresponding to each image in the support set; Using a normalized exponential function, each dimension value after multiplying the scale factor corresponding to each label class by the low-level features corresponding to each image in the query set is mapped to a target numerical range, and then added to the low-level features corresponding to each image in the query set to obtain the enhanced features corresponding to each image in the query set.

5. The method according to any one of claims 1 to 3, characterized in that The method of obtaining high-level dynamic features corresponding to each image in the support set and the query set based on the low-level features and enhanced features corresponding to each image in the support set and the query set by using a convolutional neural network including the dynamic convolution kernel parameters includes: The method further comprises: obtaining dynamic features corresponding to each image in the support set and the query set based on the low-level features corresponding to each image in the support set and the query set, using a convolutional neural network including the dynamic convolution kernel parameters; The enhanced features and dynamic features corresponding to each image in the support set are spliced ​​together to obtain the high-level features corresponding to each image in the support set, and the enhanced features and dynamic features corresponding to each image in the query set are spliced ​​together to obtain the high-level dynamic features corresponding to each image in the query set.

6. The method according to any one of claims 1 to 3, characterized in that Determining the similarity of each image in the query set corresponding to each label class based on the support set and the high-level dynamic features corresponding to each image in the query set includes: Determining label class features corresponding to each label class based on the high-level dynamic features corresponding to each image in the support set; Determining local correlation features based on the high-level dynamic features corresponding to each image in the query set and the label class features corresponding to each label class; Determining a first weight distribution feature and a second weight distribution feature based on average pooling in the horizontal direction and the numerical direction of the local correlation feature respectively; Determining the interactivity feature corresponding to each of the tag classes based on the tag class feature corresponding to each of the tag classes and the first weight distribution feature; Determining an interactivity feature corresponding to the query set based on the high-level dynamic features corresponding to each image in the query set and the second weight distribution feature; Based on the interactivity features corresponding to the tag classes and the interactivity features corresponding to the query set, the similarity between the images in the query set and the tag classes is determined.

7. The method according to claim 6, characterized in that The determining, based on the tag class features corresponding to the tag classes and the first weight distribution features, of the interactivity features corresponding to the tag classes includes: The tag class feature corresponding to each of the tag classes is multiplied by the first weight distribution feature, and then the tag class features corresponding to each of the tag classes are accumulated to obtain the interactivity feature corresponding to each of the tag classes.

8. The method according to claim 6, characterized in that Determining an interactivity feature corresponding to the query set based on the high-level dynamic features corresponding to each image in the query set and the second weight distribution feature includes: The high-level dynamic features corresponding to each image in the query set are multiplied by the second weight distribution feature, and then the high-level dynamic features corresponding to each image in the query set are accumulated to obtain the interactivity features corresponding to the query set.

9. A picture annotation device, characterized in that: include: The low-level feature extraction module is used to obtain the low-level features corresponding to each image in the support set and query set; A feature enhancement module, configured to determine a scale factor corresponding to each label class based on the low-level features corresponding to each image in the support set and the label class corresponding to each image in the support set; The feature enhancement module is further configured to perform feature enhancement on low-level features corresponding to each image in the support set and the query set based on the scale factor corresponding to each label class, thereby obtaining enhanced features corresponding to each image in the support set and the query set; A dynamic feature extraction module, configured to generate dynamic convolution kernel parameters that dynamically change with the low-level features corresponding to each image in the support set; The dynamic feature extraction module is further configured to obtain high-level features corresponding to each image in the support set and the query set based on the low-level features and enhanced features corresponding to each image in the support set and the query set, using a convolutional neural network including the dynamic convolution kernel parameters; a classification module, configured to determine a similarity between each image in the query set and each of the label classes based on high-level features corresponding to each image in the support set and the query set; The classification module is further configured to label each image in the query set based on the similarity.

10. The device according to claim 9, characterized in that The feature enhancement module is used to pool the low-level features corresponding to the images of each label class in the support set based on the label class corresponding to each image in the support set, and then use a fully connected neural network to obtain the scale factor corresponding to each label class.

11. The device according to claim 9, characterized in that The dynamic feature extraction module is used to extract the task features corresponding to the current labeling task from the low-level features corresponding to each image in the support set; The dynamic feature extraction module is further used to determine the dynamic convolution kernel parameters based on the task features corresponding to the current labeling task.

12. The device according to any one of claims 9 to 11, characterized in that The feature enhancement module is configured to use a normalized exponential function to map each dimension value obtained by multiplying the scale factor corresponding to each label class by the low-level feature corresponding to each image in the support set to a target numerical range, and then add the result to the low-level feature corresponding to each image in the support set to obtain an enhanced feature corresponding to each image in the support set; The feature enhancement module is also used to use a normalized exponential function to map each dimension value after multiplying the scale factor corresponding to each label class with the low-level feature corresponding to each image in the query set to a target numerical range, and then add it to the low-level feature corresponding to each image in the query set to obtain the enhanced feature corresponding to each image in the query set.

13. The device according to any one of claims 9 to 11, characterized in that The feature enhancement module is configured to obtain dynamic features corresponding to each image in the support set and the query set based on the low-level features corresponding to each image in the support set and the query set, using a convolutional neural network including the dynamic convolution kernel parameters; The enhanced features and dynamic features corresponding to each image in the support set are spliced ​​together to obtain the high-level features corresponding to each image in the support set, and the enhanced features and dynamic features corresponding to each image in the query set are spliced ​​together to obtain the high-level features corresponding to each image in the query set.

14. The device according to any one of claims 9 to 11, characterized in that Also includes: A mutual attention module, configured to determine label class features corresponding to each of the label classes based on the high-level dynamic features corresponding to each image in the support set; The mutual attention module is further configured to determine local correlation features based on the high-level dynamic features corresponding to each image in the query set and the label class features corresponding to each label class; The mutual attention module is further configured to determine a first weight distribution feature and a second weight distribution feature based on average pooling of the local correlation features in the horizontal direction and the numerical direction, respectively; The mutual attention module is further configured to determine an interactivity feature corresponding to each of the tag classes based on the tag class features corresponding to each of the tag classes and the first weight distribution features; The mutual attention module is further configured to determine an interactivity feature corresponding to the query set based on the high-level dynamic features corresponding to each image in the query set and the second weight distribution feature; The classification module is used to determine the similarity between each image in the query set and each tag class based on the interactivity feature corresponding to each tag class and the interactivity feature corresponding to the query set.

15. The device according to claim 14, characterized in that The mutual attention module is used to multiply the label class features corresponding to each of the label classes by the first weight distribution feature, and then accumulate the label class features corresponding to each of the label classes to obtain the interactivity features corresponding to each of the label classes.

16. The device according to claim 14, characterized in that The mutual attention module is used to multiply the high-level dynamic features corresponding to each image in the query set by the second weight distribution feature, and then accumulate the high-level dynamic features corresponding to each image in the query set to obtain the interactivity features corresponding to the query set.

17. A chip system, characterized in that: include: A memory and a processor, wherein the memory stores computer program instructions, and when the program instructions are executed by the processor, the chip system implements the method according to any one of claims 1 to 8.

18. A computer storage medium, characterized in that The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method according to any one of claims 1 to 8.

19. A computer program product, characterized in that When the computer program product is run on a computer, the computer is caused to perform the method according to any one of claims 1 to 8.

Images