Sample labeling method based on important sample mining and important sample mining method

Abstract

A sample labeling method and an important sample mining method based on important sample mining, by extracting an initial feature vector of a sample image and performing normal distribution sampling processing, the sampled feature vector conforms to the standard normal distribution, the sampled feature vector is input into a discriminator to obtain the confidence of whether the sample image is a labeled sample image, and the b unlabeled sample images with the smallest confidence are taken as important sample images, and subsequent labeling of the important sample images can be performed, thereby reducing the labeling workload of the sample image and improving the training efficiency. The method of the present application is independent of specific tasks and can be applied to various task scenarios such as classification, segmentation and target detection, and has a wide range of applications.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a sample annotation method and an important sample mining method based on important sample mining. Background Technology

[0002] In recent years, deep learning technology has advanced rapidly, playing an indelible role in technological innovation in fields such as vision and speech. However, deep learning is heavily reliant on big data, requiring a massive supply of data to optimize a vast number of parameters, enabling the model to learn how to extract high-quality features. Most deep learning tasks require large amounts of high-quality labeled datasets, but acquiring these datasets consumes significant human resources. Therefore, how to extract "high-information" or high-value data from unlabeled datasets for labeling, maximizing model performance gains while reducing labeling workload, is a significant research hotspot.

[0003] Deep learning can be divided into supervised learning, semi-supervised learning, and unsupervised learning. In the field of computer vision, supervised learning refers to training a neural network through a one-to-one correspondence between images and labeled information, enabling it to perform tasks such as classification, object detection, and semantic segmentation. Unsupervised learning only requires images and does not require additional labeled information for training.

[0004] In most scenarios, supervised and semi-supervised learning outperforms unsupervised learning. However, both supervised and semi-supervised learning require a certain amount of labeled data during training. This means that all or part of the data needs to be labeled before model training can proceed. However, obtaining labeled samples in real-world production environments is costly. On one hand, labeling images is time-consuming and labor-intensive; on the other hand, in some specific business scenarios, sample collection is difficult, requiring personnel to find ways to obtain labeled data. For example, in defect detection, the number of defective products is far less than the number of good products in industrial production, resulting in a limited number of defect images. Furthermore, the defect categories of defective products vary widely, making multiple enumerations and reproductions impossible, inevitably leading to imbalanced samples. Therefore, in these business areas, obtaining samples and building models requires human intervention. To reduce the amount of labeling, when not mining important samples, samples are usually randomly selected from the existing sample pool or selected according to manually set rules for manual labeling. While this reduces the amount of labeling, the labeling cost is always relatively high, and the training effect is not ideal. All of these factors greatly limit the verification and implementation of deep learning algorithms.

[0005] Therefore, reducing the cost of manual annotation has become a concern for practitioners, as the human and material resources required to annotate 100 samples are vastly different from those required to annotate tens of thousands of samples. It is necessary to reduce the number of samples to be annotated. Summary of the Invention

[0006] This invention provides a sample annotation method based on important sample mining, which aims to reduce the amount of annotation required for deep learning training samples.

[0007] According to the first aspect, one embodiment provides a sample annotation method based on importance sample mining, comprising:

[0008] The sample acquisition steps involve obtaining both labeled and unlabeled sample sets.

[0009] The feature encoding step involves obtaining M sample images from the labeled sample set and the unlabeled sample set, extracting features from the obtained sample images to obtain the initial feature vector of each sample image. The dimension of the initial feature vector is N, where M is the preset batch size, and M and N are both positive integers.

[0010] The feature vector sampling step involves calculating the mean vector m = (m1, m2, ..., mn) of the obtained M initial feature vectors. N ) and the standard deviation vector σ = (σ1, σ2, ..., σ N Randomly generate M N-dimensional feature vectors e j =(e j,1 ,e j,2 ,…,e j,N Each sample image corresponds to a feature vector e. j , where the eigenvector e j Each element is calculated according to the standard normal distribution, j = 1, 2, ..., M, i = 1, 2, ..., N; according to c j,i =exp(σ i )·e j,i +m i Obtain the sampled feature vector c j =(c j,1 ,c j,2 ,…,c j,N );

[0011] The feature vector decoding step utilizes the feature vector c sampled from each sample image. j Feature decoding is performed to obtain the reconstructed image, where the reconstructed target image is the original sample image;

[0012] The confidence level acquisition step involves sampling the feature vector c of each sample image. jThe input is processed by a discriminator to determine whether each sample image is a labeled sample image;

[0013] The important sample selection step involves obtaining the confidence scores of the unlabeled sample images among the M sample images, and selecting the b unlabeled sample images with the lowest confidence scores as important sample images, wherein the unlabeled sample images are images from the unlabeled sample set;

[0014] The sample update step involves labeling important sample images and adding them to the labeled sample set, and removing important sample images from the unlabeled sample set to update the labeled sample set and the unlabeled sample set. The labeled sample set is used to train the target task network.

[0015] Repeat the above steps until the preset stopping condition is met.

[0016] In one embodiment, an expert system is used to annotate important sample images.

[0017] In one embodiment, a feature encoding network is used to extract features from a sample image to obtain an initial feature vector, and each element value of the extracted initial feature vector conforms to the N(0,1) standard normal distribution. The feature encoding network consists of multiple convolutional layers, pooling layers and nonlinear units.

[0018] In one embodiment, a decoding network is used to process the sampled feature vector c. j Feature decoding is performed to obtain the reconstructed image. The decoding network consists of multiple convolutional layers, pooling layers, and nonlinear units.

[0019] In one embodiment, the sample annotation method further includes performing the following steps after the sample update step:

[0020] The sample sampling step involves randomly sampling the updated labeled sample set and the unlabeled sample set at a predetermined ratio to obtain training sample images.

[0021] The training sample processing step involves obtaining the initial feature vector, the sampled feature vector, and the confidence score obtained by inputting the sampled feature vector of the training sample image into the discriminator.

[0022] The first training step involves using the initial feature vector, sampled feature vector, and confidence level corresponding to the training sample image to train an encoder-decoder network composed of the feature encoding network and the decoding network according to a first loss function to update its parameters. The training objective is to make each element value of the initial feature vector extracted by the feature encoding network close to the standard normal distribution, to make the image reconstructed by the decoding network approximate the original training sample image, and to reduce the confidence level obtained by inputting the sampled feature vector corresponding to the labeled sample image in the training sample image into the discriminator, while increasing the confidence level obtained by inputting the sampled feature vector corresponding to the unlabeled sample image in the training sample image into the discriminator.

[0023] The second training step involves using the sampled feature vectors and confidence levels corresponding to the training sample images to train the discriminator according to the second loss function to update its parameters. The training objective is to make the discriminator output higher confidence levels for the sampled feature vectors corresponding to the labeled sample images in the training sample images, and lower confidence levels for the sampled feature vectors corresponding to the unlabeled sample images in the training sample images.

[0024] The third training step involves training the target task network using the labeled sample images from the training sample images to update the parameters of the target task network.

[0025] In one embodiment, the first loss function is derived from the training loss function. Adversarial loss function and sampling distribution loss function The training loss function is determined jointly by these three parts. The expression is:

[0026]

[0027] Where β is a preset coefficient, N1 is the number of labeled sample images in the training sample images, N2 is the number of unlabeled sample images in the training sample images, and c j,i It is the i-th element of the sampled feature vector corresponding to the j-th labeled sample image in the training sample images. This represents the i-th element c of the sampled feature vector corresponding to the j-th labeled sample image in the training sample images. j,i The probability of a corresponding pixel in the reconstructed image is obtained by inputting the decoding network; c l,i It is the i-th element of the sampled feature vector corresponding to the l-th unlabeled sample image in the training sample images. This represents the i-th element c of the sampled feature vector corresponding to the l-th unlabeled sample image in the training sample images. l,iThe probability of a pixel corresponding to the reconstructed image is obtained by inputting the decoding network; μ L,i It is the i-th element of the mean vector of the labeled sample images in the training sample images, σ L,i It is the i-th element of the standard deviation vector of the labeled sample images in the training sample images, μ U,i It is the i-th element of the mean vector of the unlabeled sample images in the training sample images, σ U,i It is the i-th element of the standard deviation vector of the unlabeled sample images in the training sample images;

[0028] The adversarial loss function The expression is:

[0029]

[0030] Where E[] represents the expected value, q φ and p θ These represent the mapping relationship between the feature encoding network and the decoding network, respectively, where D represents the discriminator, and D(q) φ (c L ,x L )) indicates that the labeled sample image x L The corresponding sampled feature vector c L Input the confidence level obtained by the discriminator, D(q) φ (c U ,x U )) indicates that the unlabeled sample image x U The corresponding sampled feature vector c U The confidence level obtained by inputting the discriminator;

[0031] The sampling distribution loss function The expression is:

[0032]

[0033] Where m i It is the i-th element of the mean vector of the training sample images, σ i It is the i-th element of the standard deviation vector of the training sample images.

[0034] In one embodiment, the expression for the first loss function is:

[0035]

[0036] Where λ1, λ2 and λ3 are preset weighting coefficients.

[0037] In one embodiment, the expression for the second loss function is:

[0038] L D =-E[logD(q) φ (c L |x L ))]-E[log(1-D(q φ (c U |x U )))];

[0039] Where E[] represents the expected value, q φ This represents the mapping relationship of the feature encoding network, where D represents the discriminator, and D(q) φ (c L |x L )) represents the labeled sample image x in the training sample image. L The corresponding sampled feature vector c L Input the confidence level obtained by the discriminator, D(q) φ (c U |x U )) represents the unlabeled sample image x in the training sample image. U The corresponding sampled feature vector c U Input the confidence level obtained from the discriminator.

[0040] According to the second aspect, one embodiment provides an important sample mining method, including:

[0041] The sample acquisition steps involve obtaining both labeled and unlabeled sample sets.

[0042] The feature encoding step involves obtaining M sample images from the labeled sample set and the unlabeled sample set, extracting features from the obtained sample images to obtain the initial feature vector of each sample image. The dimension of the initial feature vector is N, where M is the preset batch size, and M and N are both positive integers.

[0043] The feature vector sampling step involves calculating the mean vector m = (m1, m2, ..., mn) of the obtained M initial feature vectors. N ) and the standard deviation vector σ = (σ1, σ2, ..., σ N Randomly generate M N-dimensional feature vectors e j =(e j,1 ,e j,2 ,…,e j,N Each sample image corresponds to a feature vector e. j , where the eigenvector e j Each element is calculated according to the standard normal distribution, i = 1, 2, ..., N, j = 1, 2, ..., M; according to c j,i =exp(σ i)·e j,i +m i Obtain the sampled feature vector c j =(c j,1 ,c j,2 ,…,c j,N );

[0044] The feature vector decoding step utilizes the feature vector c sampled from each sample image. j Feature decoding is performed to obtain the reconstructed image, where the reconstructed target image is the original sample image;

[0045] The confidence level acquisition step involves sampling the feature vector c of each sample image. j The input is processed by a discriminator to determine whether each sample image is a labeled sample image;

[0046] The important sample selection step involves obtaining the confidence scores of the unlabeled sample images among the M sample images, and selecting the b unlabeled sample images with the lowest confidence scores as important sample images, wherein the unlabeled sample images are images from the unlabeled sample set.

[0047] According to a third aspect, one embodiment provides a computer-readable storage medium storing a program that can be executed by a processor to implement the sample annotation method as described in the first aspect and / or the important sample mining method as described in the second aspect.

[0048] The sample annotation method and important sample mining method based on important sample mining in the above embodiments extract the initial feature vector of the sample image and perform normal distribution sampling processing so that the sampled feature vector conforms to the standard normal distribution. The sampled feature vector is input into the discriminator to obtain the confidence score of whether the sample image is an labeled sample image. Then, the b unlabeled sample images with the lowest confidence scores are regarded as important sample images, and subsequent annotation of important sample images can be performed, thereby reducing the workload of sample image annotation and improving training efficiency. Since the target image reconstructed during feature decoding is the original sample image, it can guide the initial feature vector that can more effectively represent the features of the sample image during feature extraction, thereby improving the effectiveness of important sample mining. The method of the present invention is independent of specific tasks and can be applied to various task scenarios such as classification, segmentation, and object detection. In addition, the method of the present invention mines important samples based on the feature space of the sample image, rather than directly predicting the uncertainty of the sample image based on a specific task, thereby improving the adaptability of the method. Attached Figure Description

[0049] Figure 1 A schematic diagram illustrating the logical structure of the basic concept of the technical solution of this invention;

[0050] Figure 2 A flowchart of an important sample mining method in one embodiment;

[0051] Figure 3 This is a flowchart illustrating the important sample mining and annotation processes of this invention.

[0052] Figure 4 This is a confidence distribution diagram of the discriminator outputting labeled and unlabeled sample images in one embodiment.

[0053] Figure 5 A flowchart illustrating a sample annotation method based on importance sample mining, according to one embodiment;

[0054] Figure 6 A flowchart illustrating another embodiment of a sample annotation method based on importance sample mining;

[0055] Figure 7 This is a flowchart of another embodiment of a sample annotation method based on importance sample mining. Detailed Implementation

[0056] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of this application. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to this application are not shown or described in the specification. This is to avoid obscuring the core parts of this application with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0057] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0058] The serial numbers assigned to components in this document, such as "first" and "second," are used only to distinguish the described objects and have no sequential or technical meaning. The terms "connection" and "linkage" used in this application, unless otherwise specified, include both direct and indirect connections (linkages).

[0059] The current basic workflow for deep learning training is: sample labeling, training, and testing. When over-detected or under-detected samples are found, they are labeled, and the training and testing process is repeated. For detection, samples can be mainly divided into the following categories:

[0060] (1) Positive sample: The sample corresponding to the category we want to classify correctly. For example, if we need to classify an image to determine whether it belongs to a cat, then during training, images of cats are positive samples.

[0061] (2) Negative samples: According to the example above, not all other pictures of cats are negative samples;

[0062] (3) Difficult to classify positive samples: Positive samples that are misclassified as negative samples, or positive samples with the highest loss function value during training;

[0063] (4) Difficult to classify negative samples: Negative samples that are misclassified as positive samples, or negative samples with the highest loss function value during training;

[0064] (5) Easy-to-classify positive samples: positive samples that are easy to classify correctly, with a very high probability of this class, or positive samples with the lowest loss function value during training.

[0065] (6) Easy-to-classify negative samples: Negative samples that are easy to classify correctly, with a very high probability of this class, or negative samples with the lowest loss function value during training.

[0066] To reduce the difficulty of data labeling, key or important samples (often considered difficult-to-classify samples) are identified through technical means or mathematical methods and used for labeling and training, thus reducing the cost of data labeling. The general idea is to acquire those difficult-to-classify sample data through learning methods, have them manually verified and reviewed, and then use the manually labeled data to train a supervised or semi-supervised learning model, gradually improving the model's performance and integrating human experience into the model learning process.

[0067] The core idea behind current important sample mining is to use a classifier to classify samples, then add misclassified samples to a negative sample set to further train the classifier. The key is to identify training samples that negatively impact network performance and address them specifically. Simply put, it involves selecting difficult-to-classify samples, placing them in a negative sample set, and then using this to strengthen the classifier's training. However, this approach leads to an increasing number of negative samples and a decreasing number of positive samples, resulting in data imbalance.

[0068] Existing methods mainly employ two technical approaches: offline and online. When mining hard samples offline, during sample training, evaluation metrics such as IoU (Intersection over Union) and loss function are calculated between the predicted and labeled results. A threshold, such as 0.5, is typically set; samples exceeding this threshold are considered positive, and those below are considered negative, and the network is then trained using these negative samples. However, as mentioned above, this strategy may encounter a problem as training progresses: the number of positive samples may become significantly less than the number of negative samples, leading to an imbalanced data distribution and suboptimal network training results.

[0069] Another common approach is to mine hard samples online, simultaneously with training. During training, hard samples are selected for iterative processing to improve training effectiveness. This method sorts all prediction results based on their loss function values. Then, in subsequent training, a certain proportion of samples with the highest loss function values ​​are selected for backpropagation to update the model's weights. Typically, the hard sample mining part of this method is coupled with the main task network (i.e., the network trained to perform a specific task). The hard sample mining part is a component (e.g., a branch) of the main task network, and the loss function used for hard sample mining is also from the main task. However, this approach has a problem: similar prediction results tend to have similar loss function values, and the intermediate feature maps and process data from both the main task and hard sample mining tasks need to be stored for subsequent computation, resulting in significant time and space consumption.

[0070] Currently, there are some evaluation methods for determining the information content of an image, but these methods are often applied to binary or multi-class classification models. In the process of mining difficult samples, we should actually focus more on the information content at the target instance level rather than the classification information content of the entire image. This is because easy image classification does not necessarily mean easy target instance classification. If an image contains difficult-to-classify instances, it still has high value. Therefore, existing methods need to be improved to determine important samples based on the target instances of interest within the image.

[0071] The problem this invention aims to solve is to reduce the number of samples that need to be labeled. By identifying important samples in the sample images, users only need to label a small number of samples to obtain a better model.

[0072] Please refer to Figure 1The present invention is based on the following concept, comprising several key components: a labeled sample set L, a deep learning model C, an unlabeled sample set U, and an important sample mining component Q. A supervisor S may also be included to label the mined important samples. Before important sample mining, users first label a small number of samples to obtain a labeled sample set L, which is used to train the deep learning model C. The deep learning model C can be a set of data or a classifier, etc. The important sample mining component Q is used to identify samples with high information content or value from the unlabeled sample set U as important samples. S is the supervisor, who can correctly label the samples in the unlabeled sample set U; specifically, it can be an expert system, etc. The deep learning model C begins learning with a small number of initially labeled samples L. Then, the important sample mining component Q selects one or a batch of the most useful samples from the unlabeled sample set U as important samples. The supervisor S then labels the important samples, adds them to the labeled sample set L, and uses the acquired new knowledge to train the deep learning model C and perform the next round of important sample mining. The mining of important samples is a cyclical process until a certain stopping criterion is met. This criterion can be either reaching a set value for the number of iterations or reaching a set value for metrics such as accuracy.

[0073] Based on the above fundamental concept, the applicant further refined the method and proposed an important sample mining method. The method of this invention is task-independent in important sample mining; that is, it does not rely on the performance of the task that requires labeled data for training. Instead, it extracts feature vectors from sample images to construct a feature space, distinguishes labeled and unlabeled sample images based on these feature vectors, and identifies the unlabeled sample images with the greatest differences as important sample images. Many existing methods mine based on the prediction results of a specific task, requiring a segmentation or classification model to output prediction results for the sample images, and then analyzing the value of the corresponding sample images based on the quality of the prediction results. The core starting point of this invention is to analyze the value of the sample images themselves; that is, it does not analyze based on the prediction results of a particular task, but directly on the features of the sample images themselves.

[0074] Please refer to Figure 2 An embodiment of the important sample mining method of the present invention includes steps 110 to 160, wherein step 110 is a sample acquisition step, step 120 is a feature encoding step, step 130 is a feature vector sampling step, step 140 is a feature vector decoding step, step 150 is a confidence acquisition step, and step 160 is an important sample selection step. The following describes the method in conjunction with... Figure 3 Please provide a detailed explanation.

[0075] Step 110: Obtain the labeled sample set and the unlabeled sample set.

[0076] Users manually label a small number of sample images from the training network for the target task, forming a labeled sample set. The unlabeled sample images form an unlabeled sample set. Images in the labeled sample set are called labeled sample images, and images in the unlabeled sample set are called unlabeled sample images. Here, the target task refers to a specific image detection task, such as classification, segmentation, defect detection, and object detection. The target task network refers to the neural network that performs the target task.

[0077] Step 120: Obtain M sample images from the labeled sample set and the unlabeled sample set, extract features from the obtained sample images, and obtain the initial feature vector of each sample image.

[0078] Here, M is the preset batch size. Each important sample mining operation is performed in batches, acquiring a total of M sample images from the labeled and unlabeled sample sets. Then, feature extraction is performed on the acquired M sample images to obtain the initial feature vector for each sample image. The dimension of the initial feature vector is N, meaning the size of the initial feature vector is 1×N, where M and N are both positive integers.

[0079] Please refer to Figure 3 In some embodiments, feature extraction of sample images can be achieved through feature encoding. In one embodiment, a feature encoding network can be used to extract features from the sample image to obtain an initial feature vector. The feature encoding network is configured such that each element of the extracted initial feature vector conforms to an N(0,1) standard normal distribution. The feature encoding network can consist of multiple convolutional layers, pooling layers, and nonlinear units. The sample image can be downsampled before being input into the feature encoding network to reduce the amount of data processing.

[0080] Step 130: Calculate the mean vector m = (m1, m2, ..., mn) of the obtained M initial feature vectors. N ) and the standard deviation vector σ = (σ1, σ2, ..., σ N Randomly generate M N-dimensional feature vectors e j =(e j,1 ,e j,2 ,…,e j,N Each sample image corresponds to a feature vector e. j , where the eigenvector e j Each element is calculated according to the standard normal distribution, j = 1, 2, ..., M, i = 1, 2, ..., N; according to c j,i =exp(σ i )·e j,i +m i Obtain the sampled feature vector c j =(c j,1,c j,2 ,…,c j,N ).

[0081] By statistically analyzing the first element of each of the M initial feature vectors, the mean is obtained as the first element m1 of the mean vector m, the standard deviation is obtained as the first element σ1 of the standard deviation vector σ, and so on. Then, a corresponding feature vector e is generated for each sample image. j Each element e j,i Let e ​​be the probability value of randomly selecting a point on the standard normal distribution curve. For example, if the randomly selected variable is x = 1, then e j,i The probability value of the standard normal distribution when x = 1

[0082] Finally, according to c j,i =exp(σ i )·e j,i +m i Obtain the sampled feature vector corresponding to the sample image. This maps the sample image into a latent low-dimensional space, i.e., the feature space. Subsequent important sample judgments are based on the feature space, which is equivalent to directly mining important samples based on the features of the sample image itself.

[0083] The feature vector c after this step is processed j The feature vectors conform to a standard normal distribution. Since many physical quantities in the real world conform to a standard normal distribution, this step constrains the feature vectors to a standard normal distribution, which helps to extract more accurate feature vectors.

[0084] Step 140: Utilize the feature vector c after sampling from each sample image j Feature decoding is performed to obtain the reconstructed image, where the reconstructed target image is the original sample image.

[0085] The purpose of feature decoding is to reconstruct an image, and the target image to be reconstructed is the feature vector c. j The corresponding original sample image, assuming it is obtained through feature vector c j If the sample image can be reconstructed, it means that the extracted initial feature vector can effectively represent the sample image. Therefore, this step can guide the extraction of more effective initial feature vectors.

[0086] In one embodiment, a decoding network can be used to process the sampled feature vector c. j Feature decoding is performed to obtain the reconstructed image. The decoding network can consist of multiple convolutional layers, pooling layers, and nonlinear units. The decoding network and the feature encoding network in step 120 can form an encoder-decoder network, preferably an autoencoder.

[0087] Step 150: Sample the feature vector c of each sample image respectively. j The input is processed by a discriminator to determine whether each sample image is a labeled sample image. A higher confidence score indicates that the image is more likely to be a labeled sample image, while a lower confidence score indicates that the image is more likely to be an unlabeled sample image. The confidence score can be represented by a value in the range [0,1], representing probability. The discriminator can be implemented by constructing a neural network model.

[0088] Step 160: Obtain the confidence scores of the unlabeled sample images among the M sample images, and select the b unlabeled sample images with the lowest confidence scores as important sample images to complete the mining of important samples.

[0089] The core idea of ​​this invention is to encode labeled and unlabeled sample images to obtain their feature vectors, and use the feature vectors for discrimination. For an unlabeled sample image, if the difference between its feature vector and other feature vectors in the feature space is large enough, the unlabeled sample image is considered valuable and is selected as an important sample for training the target task network.

[0090] If the confidence level is within the interval [0,1], then the confidence level of the labeled sample images output by the discriminator will be close to 1, and the confidence level of the unlabeled sample images will be close to 0, with a distribution similar to... Figure 4 The curve shown is shown. Sample images with extremely low confidence are considered to be more representative data. They differ significantly from other sample images, are uncommon in the sample images, and may be anomalous data. They have greater value for training the target task network. Therefore, b unlabeled sample images with the lowest confidence are selected from the unlabeled sample images and these b sample images are considered important samples.

[0091] Based on the aforementioned important sample mining methods, this invention also provides a sample annotation method based on important sample mining. Please refer to [reference needed]. Figure 5 In one embodiment, the method includes steps 210 to 280, wherein step 210 is a sample acquisition step, step 220 is a feature encoding step, step 230 is a feature vector sampling step, step 240 is a feature vector decoding step, step 250 is a confidence acquisition step, step 260 is an important sample selection step, and step 270 is a sample update step, which will be described in detail below.

[0092] Step 210: Obtain the labeled sample set and the unlabeled sample set.

[0093] Step 220: Obtain M sample images from the labeled sample set and the unlabeled sample set, extract features from the obtained sample images, and obtain the initial feature vector of each sample image.

[0094] Step 230: Calculate the mean vector m = (m1, m2, ..., mn) of the obtained M initial feature vectors. N ) and the standard deviation vector σ = (σ1, σ2, ..., σ N Randomly generate M N-dimensional feature vectors e j =(e j,1 ,e j,2 ,…,e j,N Each sample image corresponds to a feature vector e. j , where the eigenvector e j Each element is calculated according to the standard normal distribution, j = 1, 2, ..., M, i = 1, 2, ..., N; according to c j,i =exp(σ i )·e j,i +m i Obtain the sampled feature vector c j =(c j,1 ,c j,2 ,…,c j,N ).

[0095] Step 240: Utilize the feature vector c after sampling each sample image j Feature decoding is performed to obtain the reconstructed image, where the reconstructed target image is the original sample image.

[0096] Step 250: Sample the feature vector c of each sample image respectively. j The input is processed by a discriminator to determine whether each sample image is a labeled sample image.

[0097] Step 260: Obtain the confidence scores of the unlabeled sample images among the M sample images, and select the b unlabeled sample images with the lowest confidence scores as important sample images.

[0098] Steps 210-260 are similar to steps 110-160 above and will not be repeated here.

[0099] Step 270: Label important sample images and add them to the labeled sample set, and remove important sample images from the unlabeled sample set to update the labeled and unlabeled sample sets. The labeled sample set is then used for subsequent training of the target task network.

[0100] Please refer to Figure 3 In one embodiment, an expert system can be used to annotate important sample images.

[0101] Step 280: Determine whether the preset stopping condition has been met. If so, stop the iteration; otherwise, return to step 210 to continue the iteration for mining and labeling important samples.

[0102] The preset stopping conditions can be such as the proportion of sample images used reaching a preset value or the number of iterations reaching a preset value.

[0103] In some embodiments, an encoder-decoder network is used for feature extraction and decoding. This network, along with a discriminator, forms an adversarial network used to distinguish whether a sample image is labeled or unlabeled. The encoder-decoder network and the discriminator identify valuable sample images through a minimax game. The encoder-decoder network attempts to deceive the discriminator into believing that all sample images come from the labeled sample set. Specifically, the feature vectors encoded by the encoder-decoder network need to make it difficult for the discriminator to distinguish between labeled and unlabeled sample images, resulting in similar confidence levels. Conversely, since sample images corresponding to feature vectors with significant differences in the feature space are considered important, the discriminator needs to learn to distinguish dissimilarity in the feature space, differentiating between labeled and unlabeled sample images, outputting high confidence for labeled images and low confidence for unlabeled images. This minimax game improves the performance of both the encoder-decoder network and the discriminator.

[0104] After mining important samples, an adversarial network can be trained using both labeled and unlabeled sample sets. This training enables the encoder-decoder network to compete against the discriminator. Therefore, please refer to [the relevant documentation / reference]. Figure 6 One embodiment of the sample labeling method based on important sample mining may further include steps 310 to 340 after step 270 and before step 280, wherein step 310 is a sample sampling step, step 320 is a training sample processing step, step 330 is a first training step, and step 340 is a second training step, which will be described in detail below.

[0105] Step 310: Randomly sample the updated labeled sample set and the unlabeled sample set according to a predetermined ratio to obtain training sample images. The labeled sample images in the training sample images are represented by the symbol x. L Unlabeled sample images are indicated by the symbol x. U express.

[0106] Step 320: Obtain the initial feature vector, the sampled feature vector of the training sample image, and the confidence score obtained after inputting the sampled feature vector of the training sample image into the discriminator.

[0107] For details on obtaining the initial feature vector, the sampled feature vector, and the confidence level, please refer to steps 120-150, which will not be repeated here.

[0108] Step 330: Using the initial feature vector, the sampled feature vector, and the confidence level corresponding to the training sample image, train an encoder-decoder network consisting of a feature encoding network and a decoding network according to the first loss function to update its parameters.

[0109] The training objectives are to make each element of the initial feature vector extracted by the feature encoding network approximate a standard normal distribution, to make the image reconstructed by the decoding network approximate the original training sample image, and to reduce the confidence level of the sampled feature vectors corresponding to labeled sample images in the training sample images when input into the discriminator, while increasing the confidence level of the sampled feature vectors corresponding to unlabeled sample images in the training sample images when input into the discriminator. Making the image reconstructed by the decoding network approximate the original training sample image guides the feature encoding network to extract initial feature vectors that effectively represent the sample images. Reducing the confidence level of the sampled feature vectors corresponding to labeled sample images in the training sample images when input into the discriminator, and increasing the confidence level of the sampled feature vectors corresponding to unlabeled sample images in the training sample images when input into the discriminator, achieves a minimax game between the encoder-decoder network and the discriminator, enabling the encoder-decoder network to deceive the discriminator, making it difficult to distinguish between labeled and unlabeled sample images.

[0110] Those skilled in the art can set a corresponding first loss function based on the above training objectives. This invention provides a first loss function, which is derived from the training loss function. Adversarial loss function and sampling distribution loss function The training loss function is determined jointly by these three parts. The expression is:

[0111]

[0112] Where β is a preset coefficient, N1 is the number of labeled sample images in the training sample images, N2 is the number of unlabeled sample images in the training sample images, and c j,i It is the i-th element of the sampled feature vector corresponding to the j-th labeled sample image in the training sample images. This represents the i-th element c of the sampled feature vector corresponding to the j-th labeled sample image in the training sample images. j,i The input decoding network obtains the probability of corresponding pixels in the reconstructed image; c l,i It is the i-th element of the sampled feature vector corresponding to the l-th unlabeled sample image in the training sample images. This represents the i-th element c of the sampled feature vector corresponding to the l-th unlabeled sample image in the training sample images. l,i The input decoding network obtains the probability of the corresponding pixel in the reconstructed image; μL,i It is the i-th element of the mean vector of the labeled sample images in the training sample images, σ L,i It is the i-th element of the standard deviation vector of the labeled sample images in the training sample images, μ U,i It is the i-th element of the mean vector of the unlabeled sample images in the training sample images, σ U,i It is the i-th element of the standard deviation vector of the unlabeled sample images in the training sample images.

[0113] For the sampled feature vector c (including the sampled feature vector c of the labeled sample image) j The sampled feature vector c of the unlabeled sample image l The sample image is input into a decoding network to obtain the reconstructed image. Each element in the feature vector c corresponds to a portion of the pixels in the reconstructed image. Since features are typically compressed after feature encoding, and the reconstructed image has the same resolution as the original sample image, the number of elements in the sampled feature vector c is less than the number of pixels in the reconstructed image. Therefore, each element of the feature vector c corresponds to multiple pixels in the reconstructed image. and This means that the feature vector c is input into the decoding network to obtain the reconstructed image, and each element of the feature vector c represents the probability of a corresponding pixel in the reconstructed image.

[0114] For the calculation of the mean vector and standard deviation vector, please refer to step 130. The mean vector and standard deviation vector of the labeled sample images are the initial feature vectors of all labeled sample images in the training sample images. The mean vector and standard deviation vector of the unlabeled sample images are the initial feature vectors of all unlabeled sample images in the training sample images.

[0115] Training loss function Overall, it consists of two parts, which respectively constrain labeled sample images and unlabeled sample images. During the training iteration, the image reconstructed using the feature vectors sampled from labeled and unlabeled sample images approximates the original sample image, and the feature encoding network extracts each element value of the initial feature vector from labeled and unlabeled sample images respectively conforms to a normal distribution.

[0116] Adversarial loss function The expression is:

[0117]

[0118] Where E[] represents the expected value, q φ and p θThese represent the mapping relationship between the feature encoding network and the decoding network, respectively, where D represents the discriminator, and D(q) φ (c L ,x L )) indicates that the labeled sample image x L The corresponding sampled feature vector c L The confidence level obtained from the input discriminator, D(q) φ (c U ,x U )) indicates that the unlabeled sample image x U The corresponding sampled feature vector c U The confidence level obtained from the input discriminator.

[0119] Adversarial loss function Primarily used to implement minimax game with the discriminator, through -E[logD(q φ (c L x L The part of ))] makes the labeled sample image x L The corresponding sampled feature vector c L The confidence level obtained by the input discriminator approaches 0, through -E[logD(q φ (c U ,x U The part of the unlabeled sample image x U The corresponding sampled feature vector c U The confidence level obtained after inputting the discriminator approaches 1.

[0120] Sampling distribution loss function The expression is:

[0121]

[0122] Where m i It is the i-th element of the mean vector of the training sample images, σ i It is the i-th element of the standard deviation vector of the training sample images. The mean vector and standard deviation vector of the training sample images are obtained by statistically analyzing the initial feature vectors of all training sample images.

[0123] Sampling distribution loss function It is mainly used to make the distribution of the initial feature vectors extracted by the feature encoding network closer to the standard normal distribution.

[0124] Those skilled in the art can adjust the training loss function as needed. Adversarial loss function and sampling distribution loss function The first loss function is obtained by combining the results. In one embodiment of the present invention, the expression for the first loss function is:

[0125]

[0126] λ1, λ2, and λ3 are preset weighting coefficients, which can be determined according to specific needs.

[0127] Step 340: Using the sampled feature vectors and confidence scores corresponding to the training sample images, train the discriminator according to the second loss function to update its parameters.

[0128] The training objective is to make the confidence level of the discriminator's output of the sampled feature vector corresponding to the labeled sample image in the training sample image approach 1, and the confidence level of the output of the sampled feature vector corresponding to the unlabeled sample image in the training sample image approach 0, so as to achieve a minimax game with the encoder-decoder network and learn to distinguish between labeled and unlabeled sample images.

[0129] Those skilled in the art can set a corresponding second loss function based on the above training objectives. This invention provides a second loss function, the expression of which is:

[0130] L D =-E[logD(q) φ (c L |x L ))]-E[log(1-D(q φ (c U |x U )))],

[0131] Where D(q) φ (c L |x L )) indicates that the labeled sample image x in the training sample image will be used. L The corresponding sampled feature vector c L The confidence level obtained from the input discriminator, D(q) φ (c U |x U )) indicates that the unlabeled sample image x in the training sample image will be used. U The corresponding sampled feature vector c U The confidence level obtained from the input discriminator.

[0132] Second loss function L D Primarily used to implement minimax game with encoder-decoder networks, through -E[logD(q φ (c L |x L The part that makes the discriminator target the labeled sample image x is... L The corresponding sampled feature vector c LThe output confidence level approaches 1, through -E[log(1-D(q)] φ (c U |x U The part that makes the discriminator target the unlabeled sample image x U The corresponding sampled feature vector c U The confidence level of the output approaches 0.

[0133] It should be noted that steps 310-340 can be executed multiple times before step 280 is executed, returning to step 210 for important sample mining and labeling. In other words, important sample mining and labeling can be performed after multiple rounds of training, and so on.

[0134] After identifying and labeling important samples, the labeled sample images can be used to train the target task network. Therefore, please refer to... Figure 7 One embodiment of the sample labeling method based on important sample mining further includes step 350 after step 340, which is the third training step.

[0135] Step 350: Train the target task network using the labeled sample images from the training sample images to update the parameters of the target task network. The loss function of the target task network itself is used during training.

[0136] Similarly, steps 310 to 350 can be executed multiple times before step 280 is executed, and then step 210 is returned to perform important sample mining and annotation.

[0137] The sample annotation method and important sample mining method based on important sample mining in this invention extract the initial feature vector of the sample image and perform normal distribution sampling processing, so that the sampled feature vector conforms to the standard normal distribution, which is consistent with the distribution law of most physical quantities in the real world. Then, the sampled feature vector is input into a discriminator for discrimination to obtain the confidence score of whether the sample image is an annotated sample image. The b unannotated sample images with the lowest confidence scores are then selected as important sample images, and subsequent annotation of important sample images can be performed, thereby reducing the workload of sample image annotation and improving training efficiency. Since the target image reconstructed during feature decoding is the original sample image, it can guide the initial feature vector that can more effectively represent the features of the sample image during feature extraction, improving the effectiveness of important sample mining. The method of this invention is independent of specific tasks and can be applied to various task scenarios such as classification, segmentation, and object detection.

[0138] Furthermore, the method of this invention mines important samples based on the feature space of the sample image, rather than directly predicting the uncertainty of the sample image based on a specific task, thus improving the adaptability of the method. Moreover, this invention uses the feature vector of the sample image to determine its value, and this feature vector can be reconstructed into a reconstructed image that approximates the original sample image after feature decoding. Therefore, this feature vector can reflect the information at the target instance level in the sample image, rather than just the information of the entire image, which is beneficial for mining important sample images for the target instance.

[0139] The method of this invention distinguishes between labeled and unlabeled sample images, and extracts important samples from the unlabeled sample images for labeling to train the target task network, instead of directly using them as negative samples, thus avoiding the problem of positive and negative sample imbalance.

[0140] The method of this invention decouples the important sample mining task from the target task. The important sample mining does not depend on the performance of the target task network, and the loss function is designed separately. The two are independent of each other, making it difficult to generate similar data and avoiding the consumption of storage space.

[0141] Those skilled in the art will understand that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, which may include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to achieve the above functions. For example, the program can be stored in the memory of a device, and when the program in the memory is executed by the processor, all or part of the above functions can be achieved. In addition, when all or part of the functions in the above embodiments are implemented by computer programs, the program can also be stored in a server, another computer, disk, optical disk, flash drive, or external hard drive, etc., and can be downloaded or copied to the memory of a local device, or the system of the local device can be updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be achieved.

[0142] The above examples illustrate the present invention only to aid in understanding it and are not intended to limit the scope of the invention. Those skilled in the art can make various simple deductions, modifications, or substitutions based on the principles of this invention.

Claims

1. A sample annotation method based on importance sample mining, characterized in that, include: The sample acquisition steps involve obtaining both labeled and unlabeled sample sets. The feature encoding step involves obtaining M sample images from the labeled sample set and the unlabeled sample set, extracting features from the obtained sample images to obtain the initial feature vector of each sample image. The dimension of the initial feature vector is N, where M is the preset batch size, and M and N are both positive integers. The feature vector sampling step involves calculating the mean vector m = (m1, m2, ..., mn) of the obtained M initial feature vectors. N ) and standard deviation vector σ=(σ1,σ2,,σ N Randomly generate M N-dimensional feature vectors e j =(e j,1 ,e j,2 ,,e j,N Each sample image corresponds to a feature vector e. j , where the eigenvector e j Each element is calculated according to the standard normal distribution, i = 1, 2, ..., N, j = 1, 2, ..., M; according to c j,i =exp(σ i )·e j,i +m i Obtain the sampled feature vector c j =(c j,1 ,c j,2 ,,c j,N ); The feature vector decoding step utilizes the feature vector c sampled from each sample image. j Feature decoding is performed to obtain the reconstructed image, where the reconstructed target image is the original sample image; The confidence level acquisition step involves sampling the feature vector c of each sample image. j The input is processed by a discriminator to determine whether each sample image is a labeled sample image; The important sample selection step involves obtaining the confidence scores of the unlabeled sample images among the M sample images, and selecting the b unlabeled sample images with the lowest confidence scores as important sample images, wherein the unlabeled sample images are images from the unlabeled sample set; The sample update step involves labeling important sample images and adding them to the labeled sample set, and removing important sample images from the unlabeled sample set to update the labeled sample set and the unlabeled sample set. The labeled sample set is used to train the target task network. Repeat the above steps until the preset stopping condition is met.

2. The sample labeling method as described in claim 1, characterized in that, An expert system was used to annotate important sample images.

3. The sample labeling method as described in claim 1, characterized in that, An initial feature vector is obtained by extracting features from the sample image using a feature encoding network. Each element value of the extracted initial feature vector conforms to the N(0,1) standard normal distribution. The feature encoding network consists of multiple convolutional layers, pooling layers, and nonlinear units.

4. The sample annotation method as described in claim 3, characterized in that, The sampled feature vector c is processed using a decoding network. j Feature decoding is performed to obtain the reconstructed image. The decoding network consists of multiple convolutional layers, pooling layers, and nonlinear units.

5. The sample labeling method as described in claim 4, characterized in that, This also includes performing the following steps after the sample update step: The sample sampling step involves randomly sampling the updated labeled sample set and the unlabeled sample set at a predetermined ratio to obtain training sample images. The training sample processing step involves obtaining the initial feature vector, the sampled feature vector, and the confidence score obtained by inputting the sampled feature vector of the training sample image into the discriminator. The first training step involves using the initial feature vector, sampled feature vector, and confidence level corresponding to the training sample image to train an encoder-decoder network composed of the feature encoding network and the decoding network according to a first loss function to update its parameters. The training objective is to make each element value of the initial feature vector extracted by the feature encoding network close to the standard normal distribution, to make the image reconstructed by the decoding network approximate the original training sample image, and to reduce the confidence level obtained by inputting the sampled feature vector corresponding to the labeled sample image in the training sample image into the discriminator, while increasing the confidence level obtained by inputting the sampled feature vector corresponding to the unlabeled sample image in the training sample image into the discriminator. The second training step involves using the sampled feature vectors and confidence levels corresponding to the training sample images to train the discriminator according to the second loss function to update its parameters. The training objective is to make the confidence level output by the discriminator for the sampled feature vectors corresponding to the labeled sample images in the training sample images approach 1, and the confidence level output for the sampled feature vectors corresponding to the unlabeled sample images in the training sample images approach 0. The third training step involves training the target task network using the labeled sample images from the training sample images to update the parameters of the target task network.

6. The sample labeling method as described in claim 5, characterized in that, The first loss function is derived from the training loss function. Adversarial loss function and sampling distribution loss function The training loss function is determined jointly by these three parts. The expression is: Where β is a preset coefficient, N1 is the number of labeled sample images in the training sample images, N2 is the number of unlabeled sample images in the training sample images, and c j,i It is the i-th element of the sampled feature vector corresponding to the j-th labeled sample image in the training sample images. This represents the i-th element c of the sampled feature vector corresponding to the j-th labeled sample image in the training sample images. j,i The probability of a corresponding pixel in the reconstructed image is obtained by inputting the decoding network; c l,i It is the i-th element of the sampled feature vector corresponding to the l-th unlabeled sample image in the training sample images. This represents the i-th element c of the sampled feature vector corresponding to the l-th unlabeled sample image in the training sample images. l,i The probability of a pixel corresponding to the reconstructed image is obtained by inputting the decoding network; μ L,i It is the i-th element of the mean vector of the labeled sample images in the training sample images, σ L,i It is the i-th element of the standard deviation vector of the labeled sample images in the training sample images, μ U,i It is the i-th element of the mean vector of the unlabeled sample images in the training sample images, σ U,i It is the i-th element of the standard deviation vector of the unlabeled sample images in the training sample images; The adversarial loss function The expression is: Where E[] represents the expected value, q φ This represents the mapping relationship of the feature encoding network, where D represents the discriminator, and D(q) φ (c L ,x L )) indicates that the labeled sample image x L The corresponding sampled feature vector c L Input the confidence level obtained by the discriminator, D(q) φ (c U ,x U )) indicates that the unlabeled sample image x U The corresponding sampled feature vector c U The confidence level obtained by inputting the discriminator; The sampling distribution loss function The expression is: Where m i It is the i-th element of the mean vector of the training sample images, σ i It is the i-th element of the standard deviation vector of the training sample images.

7. The sample labeling method as described in claim 6, characterized in that, The expression for the first loss function is: Where λ1, λ2 and λ3 are preset weighting coefficients.

8. The sample labeling method as described in claim 5, characterized in that, The expression for the second loss function is: L D =-E[logD(q φ (c L |x L ))]-E[log(1-D(q φ (c U |x U )))]; Where E[] represents the expected value, q φ This represents the mapping relationship of the feature encoding network, where D represents the discriminator, and D(q) φ (c L |x L )) represents the labeled sample image x in the training sample image. L The corresponding sampled feature vector c L Input the confidence level obtained by the discriminator, D(q) φ (c U |x U )) represents the unlabeled sample image x in the training sample image. U The corresponding sampled feature vector c U Input the confidence level obtained from the discriminator.

9. An important sample mining method, characterized in that, include: The sample acquisition steps involve obtaining both labeled and unlabeled sample sets. The feature encoding step involves obtaining M sample images from the labeled sample set and the unlabeled sample set, extracting features from the obtained sample images to obtain the initial feature vector of each sample image. The dimension of the initial feature vector is N, where M is the preset batch size, and M and N are both positive integers. The feature vector sampling step involves calculating the mean vector m = (m1, m2, ..., mn) of the obtained M initial feature vectors. N ) and standard deviation vector σ=(σ1,σ2,,σ N Randomly generate M N-dimensional feature vectors e j =(e j,1 ,e j,2 ,,e j,N Each sample image corresponds to a feature vector e. j , where the eigenvector e j Each element is calculated according to the standard normal distribution, i = 1, 2, ..., N, j = 1, 2, ..., M; according to c j,i =exp(σ i )·e j,i +m i Obtain the sampled feature vector c j =(c j,1 ,c j,2 ,,c j,N ); The feature vector decoding step utilizes the feature vector c sampled from each sample image. j Feature decoding is performed to obtain the reconstructed image, where the reconstructed target image is the original sample image; The confidence level acquisition step involves sampling the feature vector c of each sample image. j The input is processed by a discriminator to determine whether each sample image is a labeled sample image; The important sample selection step involves obtaining the confidence scores of the unlabeled sample images among the M sample images, and selecting the b unlabeled sample images with the lowest confidence scores as important sample images, wherein the unlabeled sample images are images from the unlabeled sample set.

10. A computer-readable storage medium, characterized in that, The medium stores a program that can be executed by a processor to implement the sample annotation method as described in any one of claims 1 to 8 and / or the important sample mining method as described in claim 9.

Images