Model training method, device and equipment based on diversity difficult negative sample mining

Abstract

The invention relates to a model training method, device and equipment based on diversity difficult negative sample mining. The method comprises the steps of evaluating a diversity score corresponding to each negative sample in a negative sample set; enabling the training batch t to be equal to 0, and enabling the 0th training set to be equal to the positive sample set; optimizing parameters of the to-be-trained model through the t-th training set to obtain a t-th to-be-trained model; inputting the negative sample set into the t-th to-be-trained model, and obtaining a probability value that each negative sample in the negative sample set belongs to the positive sample and a corresponding prediction label; screening a difficult sample set according to the probability value of each negative sample belonging to the positive sample and the prediction label; according to the diversity score and the difficult sample set, calculating a comprehensive score corresponding to each negative sample in the negative sample set; screening a tth difficult negative sample data set according to the comprehensive score; merging the t-th difficult negative sample data set and the positive sample set to obtain a (t + 1)-th training set; enabling the training batch t to be equal to t + 1, and repeatedly optimizing the parameters of the to-be-trained model through the t-th training set.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model training method, device and equipment based on diverse difficult negative sample mining. Background Art

[0002] Machine vision is a rapidly developing branch of artificial intelligence. It uses an image acquisition device (such as a camera) to obtain images of the target to be detected, and then uses an artificial intelligence-based image recognition model to extract features and classify the images of the target to be detected, and finally identifies the relevant information of the target to be detected. When machine vision technology is applied to the field of industrial product defect detection, it can automatically identify whether the product has defects based on the image of the product to be detected. It has an accuracy not lower than that of human eye recognition and a very high detection speed, and almost no wear and tear on the product. Based on the above advantages, machine vision technology is currently widely used in the field of industrial product defect detection.

[0003] The core of machine vision technology is an image recognition model based on artificial intelligence. When a sample image with a real label is input into the image recognition model, the image recognition model will output the predicted label of the sample image. If the real label is T (true), the predicted label is also T (true), and the sample image is recorded as a true sample (True Positives, TP); if the real label is F (false), the predicted label is T (true), and the sample image is recorded as a false positive sample (False Positives, FP); if the real label is T (true), the predicted label is F (false), and the sample image is recorded as a false negative sample (False Negatives, FN); if the real label is F (false), the predicted label is also F (false), and the sample image is recorded as a true negative sample (True Negatives, TN). In the field of industrial product defect detection, a label of T (true) indicates that a defect exists, and a label of F (false) indicates that a defect does not exist.

[0004] Difficult negative samples refer to sample images that are highly similar to positive samples and are easily misidentified as false positive samples (FP) by the model, but are actually negative samples. Difficult negative sample mining refers to the special mining and processing of difficult negative samples when training image recognition models to improve the effect of model training and the recognition performance of the image recognition model. Its principle is similar to that of humans picking out wrong questions in homework to form a "wrong question book" when learning, and consolidating weak knowledge by reviewing the contents of the "wrong question book".

[0005] See also Figure 1 , Figure 1The figure is a flowchart of an existing online difficult sample mining method. The OHEM (Online Hard Example Mining) method is an existing difficult negative sample mining method, which includes the following steps: S01, copying the image recognition model to be trained into the same first image recognition sub-model and second image recognition sub-model; S02, inputting multiple sample images into the first image recognition sub-model, and the first image recognition sub-model outputs predicted labels; calculating the loss function by comparing the predicted labels with the true labels; screening out several false positive samples with the largest loss function values ​​to form a difficult negative sample data set; S03, inputting the difficult negative sample data set into the second image recognition sub-model as a training set, optimizing the parameters of the second image recognition sub-model; using the optimized second image recognition sub-model as a new first image recognition sub-model, and returning to step S02; S04, repeating steps S02-S03, iteratively optimizing the parameters of the second image recognition sub-model, until the training is terminated when the preset end condition is met, and using the final second image recognition sub-model as the trained image recognition model.

[0006] The existing OHEM method has the following defects: 1. It only selects difficult negative samples based on the numerical value of the loss function, which cannot guarantee the diversity and typicality of the difficult negative samples selected, and has a limited effect on improving the model training effect. 2. Taking improving the prediction accuracy of the model as the training goal, the trained image recognition model is too cautious in order to avoid misjudgment, resulting in stricter standards for judging the existence of defects, which is prone to missed detection (more false negative samples). Summary of the invention

[0007] Based on this, the purpose of the present invention is to provide a model training method, device and electronic device based on diverse difficult negative sample mining, which comprehensively considers the two indicators of diversity and difficulty to screen difficult negative samples, ensures that the screened difficult negative samples are diverse and typical, effectively improves the training effect of the model, and enables the model to achieve a higher recall rate.

[0008] The present invention provides a model training method based on diversity difficult negative sample mining, which comprises the following steps: S0, obtaining a positive sample set and a negative sample set; S1, evaluating the diversity score corresponding to each negative sample in the negative sample set through a diversity evaluation network; S2, setting the training batch t=0, setting the 0th training set equal to the positive sample set; S3, optimizing the parameters of the model to be trained through the tth training set to obtain the tth model to be trained; inputting the negative sample set into the tth model to be trained, and the tth model to be trained outputs the probability value of each negative sample in the negative sample set belonging to the positive sample and the corresponding prediction label; S4, according to each The probability value of the negative sample belonging to the positive sample and the predicted label are used to filter the difficult sample set; S5, according to the diversity score and the difficult sample set, the comprehensive score corresponding to each negative sample in the negative sample set is calculated; S6, according to the comprehensive score, the t-th difficult negative sample data set is filtered; S7, the t-th difficult negative sample data set and the positive sample set are merged to obtain the t+1-th training set; S8, the training batch t=t+1, and steps S3-S7 are repeated to repeatedly optimize the parameters of the model to be trained; until the preset first end condition is met, the training is terminated, and the final t-th model to be trained is used as the trained model to be trained.

[0009] The present invention has the following technical effects: 1. The present invention sets a diversity evaluation network to evaluate the diversity of sample images, and comprehensively considers the two indicators of diversity and difficulty to screen difficult negative samples, ensuring that the screened difficult negative sample data set can cover a variety of typical errors, improving the quality of the difficult negative sample data set, improving the effect of difficult sample mining, and greatly improving the model performance. 2. Since false negative samples are actually positive samples, and the present invention allows the positive sample set to be added unconditionally when screening the training set, it ensures that all false negative samples are included in the training set of each batch, thereby strengthening the learning of false negative samples by the model to be trained, effectively and efficiently improving the recall rate of the model, and adapting to the needs of the field of industrial product defect detection. 3. The present invention does not need to change the network structure of the model to be trained, nor does it need to obtain more original training data. It only needs to iteratively change the components of the model training set to effectively improve the training effect of the model, and has a high cost performance.

[0010] Furthermore, the diversity evaluation network includes an encoder and a discriminator; step S1 specifically includes: S11, extracting features of negative samples through the encoder to obtain low-dimensional potential features; S12, predicting the probability that the negative samples belong to positive samples based on the potential low-dimensional features through the discriminator; S13, mapping the probability that the negative samples belong to positive samples to a diversity score.

[0011] Furthermore, in step S13, the diversity score D s The calculation formula is: s ={1-Φ D(I)|I∈D neg}, where D neg represents the negative sample set, I represents the negative sample in the negative sample set, Φ D (I) represents the probability that the negative sample predicted by the discriminator belongs to the positive sample.

[0012] Furthermore, the diversity evaluation network also includes a generator; the training method for the diversity evaluation network includes the following steps: ST1, obtaining a training set including a number of positive samples and negative samples, and inputting the samples in the training set into the encoder one by one; the encoder extracts features from each sample in the training set and outputs low-dimensional potential features corresponding to each sample; ST2, the generator generates a reconstructed input sample based on the low-dimensional potential features extracted by the encoder; ST3, comparing the difference between the reconstructed input sample and the real input sample, and calculating the reconstruction loss function; ST4, the discriminator predicts the probability that each sample in the training set belongs to a positive sample based on the low-dimensional potential features extracted by the encoder; ST5, letting the positive samples in the training set The probability label value of the sample is 1, and the probability label value of the negative sample is 0; compare the probability that each sample predicted by the discriminator belongs to the positive sample and the difference between the probability label value, and calculate the discriminator loss function; ST6, let the adversarial label value of each sample in the training set be 1; compare the probability that each sample predicted by the discriminator belongs to the positive sample and the difference between the adversarial label value, and calculate the adversarial loss function; ST7, calculate the encoder loss function according to the reconstruction loss function and the adversarial loss function; ST8, iteratively optimize the parameters of the encoder according to the encoder loss function, and iteratively optimize the parameters of the discriminator according to the discriminator loss function; until the preset second end condition is met, the training is terminated to obtain a trained diversity evaluation network.

[0013] Furthermore, the reconstruction loss function L recon The calculation formula is: Where θ AE represents the parameters of the encoder, I i represents the i-th real input sample, represents the i-th reconstructed input sample, MSE() represents the mean square error, and M represents the total number of samples;

[0014] The discriminator loss function L dis The calculation formula is: Where θ dis represents the parameters of the discriminator, y i dis represents the probability label value of the i-th sample, represents the probability that the i-th sample predicted by the discriminator belongs to a positive sample, BCE() represents binary cross entropy, and M represents the total number of samples;

[0015] The adversarial loss function L adv The calculation formula is: Where θ AE represents the parameters of the encoder, y i adv represents the adversarial label value of the i-th sample, represents the probability that the i-th sample predicted by the discriminator belongs to a positive sample, BCE() represents binary cross entropy, and M represents the total number of samples;

[0016] The encoder loss function L AE The calculation formula is: L AE (θ AE )=L recon +λ·L adv , where L recon represents the reconstruction loss function, L adv represents the adversarial loss function, and λ is a pre-set adversarial coefficient.

[0017] Further, step S4 specifically includes: S41, screening negative samples whose predicted labels are not equal to the true labels to form a first difficult sample set; the first difficult sample set is screened by the following formula: Where H t E represents the first difficult sample set, Ⅱ() represents the indicator function, Represents the predicted label output by the model to be trained, y T Indicates the true label of the negative sample; S42, according to the probability value of each negative sample belonging to the positive sample, calculate the probability interval, and screen out the negative samples whose probability interval is less than the preset interval threshold to form a second difficult sample set; the second difficult sample set is screened by the following formula: H t M ={Ⅱ(2p t I -1|<τ)I∈D neg}, where H t M represents the second difficult sample set, Ⅱ() represents the indicator function, p t I represents the probability value of a negative sample belonging to a positive sample, and τ represents a preset interval threshold; S43, obtaining a joint difficult sample set according to the first difficult sample set and the second difficult sample set; the joint difficult sample set is screened by the following formula: H t =H t E ∨H t M , where H t represents the joint difficult sample set, H tE represents the first difficult sample set, H t M represents the second difficult sample set, and ∨ represents the element-wise OR operation.

[0018] Further, step S5 is specifically as follows: according to the diversity score D s and difficult sample set H t , calculate the comprehensive score V corresponding to each negative sample in the negative sample set t , the comprehensive score V t Calculated by the following formula: V t =H t ·D s , where V t Indicates the comprehensive score, H t Denotes the difficult sample set, D s Represents the diversity score.

[0019] Further, step S6 is specifically as follows: screening out the comprehensive score V t The top K negative samples are used as the t-th difficult negative sample dataset D t neg , that is, the t-th difficult negative sample dataset D t neg Filter by the following formula: D t neg =top(V t ,K,D neg ), K is the preset capacity of the difficult negative sample dataset.

[0020] Based on the same inventive concept, the present invention also provides a model training device based on diversity difficult negative sample mining, which includes: a sample acquisition module, which is used to acquire positive sample sets and negative sample sets; a diversity evaluation network, which is used to evaluate the diversity score corresponding to each negative sample in the negative sample set; an initialization module, which is used to set the training batch t=0, and set the 0th training set equal to the positive sample set; an optimization module, which is used to optimize the parameters of the model to be trained through the tth training set to obtain the tth model to be trained; the negative sample set is input into the tth model to be trained, and the tth model to be trained outputs the probability value of each negative sample in the negative sample set belonging to the positive sample and the corresponding prediction label; a difficulty screening module, which is used to select the probability value of each negative sample belonging to the positive sample and the prediction label according to the probability value of each negative sample belonging to the positive sample and the prediction label; a test label to screen the difficult sample set; a comprehensive scoring module, which is used to calculate the comprehensive score corresponding to each negative sample in the negative sample set according to the diversity score and the difficult sample set; a difficult negative sample screening module, which is used to screen the t-th difficult negative sample data set according to the comprehensive score; a training set merging module, which is used to merge the t-th difficult negative sample data set and the positive sample set to obtain the t+1-th training set; an iteration module, which is used to set the training batch t=t+1, call the optimization module, the difficulty screening module, the comprehensive scoring module, the difficult negative sample screening module and the training set merging module, and repeatedly optimize the parameters of the model to be trained; until the preset first end condition is met, the training is terminated, and the final t-th model to be trained is used as the trained model to be trained.

[0021] Based on the same inventive concept, the present invention also provides an electronic device, comprising: a processor; a memory for storing a computer program executed by the processor; wherein, when the processor executes the computer program, it implements any one of the above-mentioned model training methods based on diverse difficult negative sample mining.

[0022] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flowchart of the existing online difficult sample mining methods;

[0024] Figure 2 It is a module schematic diagram of a model training device based on diverse difficult negative sample mining of the present invention;

[0025] Figure 3 It is a flow chart of the model training method based on diverse difficult negative sample mining of the present invention;

[0026] Figure 4 A schematic diagram of the structure of a diversity evaluation network in one embodiment of the present invention;

[0027] Figure 5The figure is a flowchart of a method for training a diversity assessment network in one embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following embodiments of the technical solution of the present application are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0030] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0031] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0032] The present invention creatively designs a diversity evaluation network to evaluate the diversity of sample images, and comprehensively considers the two indicators of diversity and difficulty to screen difficult negative samples, thereby ensuring that the screened difficult negative sample data set can cover a variety of error typicalities, improving the quality of the difficult negative sample data set, and thus improving the training effect of the model.

[0033] The indicators for evaluating the performance of the image recognition model mainly include accuracy, precision, recall rate and F1 value, where the calculation formula of recall rate is: It can be seen that the recall rate is equal to the coverage rate of correctly identified positive samples to all positive samples. In the field of industrial product defect detection, the loss caused by the image recognition model misidentifying a negative sample (non-defective) as a positive sample (defective) is much smaller than the loss caused by misidentifying a positive sample (defective) as a negative sample (non-defective). The higher the recall rate of the image recognition model, the lower the probability that the image recognition model misidentifies a positive sample as a negative sample. When screening the training set, the present invention ensures that all false negative samples are included in the training set of each batch, thereby strengthening the learning of false negative samples by the model to be trained, improving the recall rate of the model, and better adapting to the needs of the field of industrial product defect detection.

[0034] See also Figure 2-3 , Figure 2 Schematic diagram of a module of a model training device based on diverse difficult negative sample mining according to the present invention. Figure 3 It is a flow chart of the model training method based on diverse and hard negative sample mining (Diverse and Hard Negatives Mining, DHNM) of the present invention.

[0035] The model training device based on diversity difficult negative sample mining of the present invention is used to train a model to be trained, which includes: a sample acquisition module 0, a diversity evaluation network 1, an initialization module 2, an optimization module 3, a difficulty screening module 4, a comprehensive scoring module 5, a difficult negative sample screening module 6, a training set merging module 7 and an iteration module 8.

[0036] The sample acquisition module 0 is used to execute step S0: obtain a positive sample set D pos And negative sample set D neg .

[0037] The diversity evaluation network 1 is used to perform step S1: evaluate the negative sample set D neg The diversity score D corresponding to each negative sample in s .

[0038] In this embodiment, since the model to be trained is an image recognition model for defect detection, the positive sample set D pos The negative sample set D includes several positive sample images with the real label T (defective, value = 1). neg The training model includes several negative sample images whose true label is F (no defect, value = 0). In other embodiments, the training model can be other classification recognition models, and the present invention does not limit the specific use of the training model.

[0039] The diversity evaluation network 1 is used to evaluate the diversity score corresponding to each negative sample in the negative sample set. The higher the diversity score of the negative sample, the greater the difference between the negative sample and the positive sample, which helps the model to be trained to grasp the difference between the positive sample and the negative sample and identify a wider range of defect-free patterns.

[0040] In one embodiment, the diversity assessment network 1 may include: a feature extraction unit, a cosine distance calculation unit and a mapping unit. The feature extraction unit (such as VGGNet or autoencoder) performs feature extraction on each sample image to obtain a feature vector. The cosine distance calculation unit is used to calculate the cosine distance between the feature vector corresponding to each negative sample and the feature vector corresponding to each positive sample, thereby measuring the difference between each negative sample and the positive sample. The mapping unit maps the cosine distance to a diversity score, so that the greater the difference between the negative sample and the positive sample, the higher the diversity score. This diversity assessment method based on cosine distance needs to calculate the cosine distance between the feature vector corresponding to each negative sample and the feature vector corresponding to each positive sample, which has a large amount of calculation and takes a long time; and the cosine distance only considers the image content and lacks information related to specific tasks.

[0041] See also Figure 4 , Figure 4 Schematic diagram of the structure of a diversity evaluation network in an embodiment of the present invention. In this embodiment, the diversity evaluation network 1 includes: an encoder 11, a discriminator 12, a mapping module 13 and a generator 14. The encoder 11 includes: a plurality of ConvBlock layers and a fully connected layer cascaded in sequence; each ConvBlock layer includes a convolutional layer, a BN layer and a ReLU layer cascaded in sequence. The discriminator 12 includes: two FCBlock layers, a fully connected layer and a Sigmoid layer cascaded in sequence; each FCBlock layer includes a fully connected layer, a BN layer and a ReLU layer cascaded in sequence. The generator 14 includes: a fully connected layer and a plurality of TConvBlock layers cascaded in sequence; each TConvBlock layer includes a transposed convolutional layer, a BN layer and a ReLU layer cascaded in sequence.

[0042] The encoder 11 is used to execute step S11: extract features from the input sample I to obtain low-dimensional potential features z. The discriminator 12 is used to execute step S12: predict the probability Φ of the input sample I being a positive sample based on the potential low-dimensional features z. D (I). The mapping module 13 is used to execute step S13: convert the probability Φ of the input sample I belonging to a positive sample D (I) Mapping to diversity score D s, thus ensuring that the greater the difference between negative samples and positive samples, the greater the diversity score D s The mapping module calculates the diversity score D s The formula is:

[0043] D s ={1-Φ D (I)|I∈D neg}, where D neg represents the negative sample set, I represents the negative sample in the negative sample set, Φ D (I) represents the probability that the negative sample predicted by the discriminator belongs to the positive sample.

[0044] The generator 14 is used to generate a reconstructed input sample according to the low-dimensional potential feature z. The reconstructed input sample The true label of is the same as the true label of the input sample I. The reconstructed input sample generated by the generator 14 Used to perform adversarial training on the encoder 11 and the discriminator 12.

[0045] See also Figure 5 , Figure 5 FIG. 1 is a flow chart of a training method for a diversity evaluation network in one embodiment of the present invention. The training method for performing adversarial training on the encoder 11 and the discriminator 12 (i.e., the training method for the diversity evaluation network) includes:

[0046] ST1, obtain a training set including several positive samples and negative samples, and input the samples in the training set into the encoder one by one; the encoder extracts features from each sample in the training set and outputs a low-dimensional potential feature z corresponding to each sample.

[0047] ST2, the generator generates a reconstructed input sample based on the low-dimensional potential feature z extracted by the encoder

[0048] ST3, compare the reconstructed input samples The difference between the real input sample I and the reconstruction loss function L is calculated recon , the calculation formula of the reconstruction loss function is:

[0049] Where θ AE represents the parameters of the encoder, I i represents the i-th real input sample, represents the i-th reconstructed input sample, MSE() represents the mean square error, and M represents the total number of samples.

[0050] ST4, the discriminator predicts the probability that each sample in the training set is a positive sample based on the low-dimensional potential features extracted by the encoder.

[0051] ST5, let the probability label value of the positive sample in the training set be 1, and the probability label value of the negative sample be 0; compare the probability that each sample predicted by the discriminator belongs to the positive sample with the difference between the probability label value, and calculate the discriminator loss function L dis , the discriminator loss function L dis The calculation formula is:

[0052] Where θ dis represents the parameters of the discriminator, y i dis represents the probability label value of the i-th sample (positive sample is 1, negative sample is 0), represents the probability that the i-th sample predicted by the discriminator belongs to a positive sample, BCE() represents binary cross entropy, and M represents the total number of samples.

[0053] ST6, set the adversarial label value of each sample in the training set to 1; compare the probability that each sample predicted by the discriminator belongs to a positive sample with the difference between the adversarial label value, and calculate the adversarial loss function L adv , the adversarial loss function L adv The calculation formula is:

[0054] Where θ AE represents the parameters of the encoder, y i adv represents the adversarial label value of the i-th sample, represents the probability that the i-th sample predicted by the discriminator belongs to a positive sample, BCE() represents binary cross entropy, and M represents the total number of samples.

[0055] ST7, according to the reconstruction loss function L recon And the adversarial loss function L adv , calculate the encoder loss function L AE , the encoder loss function L AE The calculation formula is:

[0056] L AE (θ AE )=L recon +λ·L adv , where L recon represents the reconstruction loss function, L advDenotes the adversarial loss function, and λ is a pre-set adversarial coefficient (hyperparameter). In this embodiment, in order to prevent the low-dimensional potential feature z extracted by the encoder from being unable to effectively represent the input image, the adversarial coefficient λ is set to 0 in the first several training batches.

[0057] ST8, according to the encoder loss function L AE Iteratively optimize the parameters of the encoder according to the discriminator loss function L dis Iteratively optimize the parameters of the discriminator; until a preset second end condition is met, the training is terminated to obtain a trained diversity evaluation network (i.e., a trained encoder and discriminator). In this embodiment, the preset second end condition is: the number of parameter iterations of the encoder and discriminator reaches a preset second number threshold or the parameters of the encoder and discriminator converge.

[0058] The initialization module 2 is used to execute step S2: record the training batch t=0, set the 0th training set D0 equal to the positive sample set D pos .

[0059] The optimization module 3 is used to execute step S3: transform the tth training set D t Input into the model to be trained, optimize the parameters of the model to be trained, and obtain the tth model to be trained Φ t T ; Set the negative sample set D neg Input to the tth model to be trained Φ t T In the example, the tth model to be trained Φ t T Output the probability value of each negative sample in the negative sample set belonging to the positive sample and the corresponding predicted label.

[0060] Specifically, the probability value that the negative sample output by the training model belongs to the positive sample When this probability value , the corresponding prediction label is T (value = 1), indicating that the model to be trained determines that the negative sample (no defect) belongs to the positive sample (defective); when this probability value , the corresponding prediction label is F (value = 0), indicating that the to-be-trained model determines that the negative sample (no defect) does not belong to the positive sample (defective).

[0061] The difficulty screening module 4 is used to perform step S4: screening the difficult sample set H according to the probability value of each negative sample belonging to the positive sample and the predicted label. t .

[0062] Step S4 includes the following steps:

[0063] S41, screening negative samples whose predicted labels are not equal to the true labels to form a first difficult sample set; the first difficult sample set is screened by the following formula:

[0064] Where H t E represents the first difficult sample set, Ⅱ() represents the indicator function, Represents the predicted label output by the model to be trained, y T represents the true label of the negative sample (value = 0). The first difficult sample set H t E In , negative samples that are screened as difficult samples are marked as 1, and other simple samples are marked as 0.

[0065] S42, according to the probability value of each negative sample belonging to the positive sample, the probability interval is calculated, and the negative samples whose probability interval is less than the preset interval threshold are screened out to form a second difficult sample set; the second difficult sample set is screened by the following formula:

[0066] H t M ={Ⅱ(|2p t I -1|<τ)|I∈D neg}, where H t M represents the second difficult sample set, Ⅱ() represents the indicator function, p t I represents the probability value of the negative sample belonging to the positive sample, and τ represents the preset interval threshold. The second difficult sample set H t M In , negative samples that are screened as difficult samples are marked as 1, and other simple samples are marked as 0.

[0067] The probability interval refers to the probability value p of the trained model predicting that a sample belongs to a positive sample. t I , and the difference between the probability value of predicting that the sample belongs to a negative sample, which is equal to |2p t I -1|. A smaller probability interval means that the model to be trained is less confident in the correctness of the prediction and the prediction is more difficult.

[0068] S43, obtaining a combined difficult sample set according to the first difficult sample set and the second difficult sample set; the combined difficult sample set is screened by the following formula:

[0069] H t =H t E ∨H t M , where Ht represents the joint difficult sample set, H t E represents the first difficult sample set, H t M Denotes the second difficult sample set, ∨ denotes the element-by-element OR operation. t In , negative samples that are screened as difficult samples are marked as 1, and other simple samples are marked as 0.

[0070] The comprehensive scoring module 5 is used to execute step S5: according to the diversity score D s and difficult sample set H t , calculate the comprehensive score V corresponding to each negative sample in the negative sample set t , the comprehensive score V t Calculated by the following formula:

[0071] V t =H t ·D s , where V t Indicates the comprehensive score, H t Denotes the difficult sample set, D s Represents the diversity score.

[0072] The difficult negative sample screening module 6 is used to perform step S6: screening out the comprehensive score V t The top K negative samples are used as the t-th difficult negative sample dataset D t neg , that is, the t-th difficult negative sample dataset D t neg Filter by the following formula: D t neg =top(V t ,K,D neg ).

[0073] The training set merging module 7 is used to execute step S7: let the t-th difficult negative sample data set D t neg and positive sample set D pos Merge to get the t+1th training set D t+1 .

[0074] The iteration module 8 is used to execute step S8: let the training batch t=t+1, call the optimization module, the difficulty screening module, the comprehensive scoring module, the difficult negative sample screening module and the training set merging module to repeat steps S3-S8 to repeatedly optimize the parameters of the model to be trained; until the preset first end condition is met, the training is terminated, and the final tth model to be trained Φ t TThe training is completed as the output of the to-be-trained model. In this embodiment, the preset first end condition is: the training batch t reaches a preset first number threshold or the parameters of the to-be-trained model converge.

[0075] The present invention has the following technical effects: 1. The present invention creatively designs a diversity evaluation network to evaluate the diversity of sample images, and comprehensively considers the two indicators of diversity and difficulty to screen difficult negative samples, ensuring that the screened difficult negative sample data set can cover a variety of typical errors, improving the quality of the difficult negative sample data set, improving the effect of difficult sample mining, and greatly improving the model performance. 2. Since false negative samples (FN) are actually positive samples, and the present invention allows the positive sample set to be unconditionally added when screening the training set, it is ensured that all false negative samples are included in the training set of each batch, thereby strengthening the learning of false negative samples by the model to be trained, effectively and efficiently improving the recall rate of the model, and meeting the needs of the field of industrial product defect detection. 3. The present invention does not need to change the network structure of the model to be trained, nor does it need to obtain more original training data. It only needs to iteratively change the components of the model training set to effectively improve the training effect of the model, and has a high cost-effectiveness. Experimental results show that the model training method based on diverse difficult negative sample mining proposed by the present invention has a good performance in P 0.99 The performance of YOLO X and YOLO v4 has been improved by more than 13% in terms of indicators. At the same time, similar performance improvements have been observed on the public KolektorSDD2 dataset, specifically P 0.99 The indicator grew by about 5%.

[0076] Based on the same inventive concept, the present application also provides an electronic device, which may be a terminal device such as a server, a desktop computing device or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). The device includes one or more processors and a memory, wherein the processor is used to execute a program to implement the model training method based on diverse difficult negative sample mining according to an embodiment of the present invention; and the memory is used to store a computer program executable by the processor.

[0077] Based on the same inventive concept, the present application also provides a computer-readable storage medium, corresponding to the embodiment of the aforementioned model training method based on diverse difficult negative sample mining, and the computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the model training method based on diverse difficult negative sample mining recorded in any of the above embodiments.

[0078] The present application may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0079] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, and the present invention is also intended to include these modifications and modifications.

Claims

1. A model training method based on diverse difficult negative sample mining, characterized in that: Includes steps: S0, obtain positive sample set and negative sample set; S1, through a diversity evaluation network, evaluates the diversity score corresponding to each negative sample in the negative sample set; S2, let training batch t = 0, let the 0th training set equal the positive sample set; S3, optimizing the parameters of the model to be trained by the tth training set to obtain the tth model to be trained; inputting the negative sample set into the tth model to be trained, and the tth model to be trained outputs the probability value of each negative sample in the negative sample set belonging to the positive sample and the corresponding prediction label; S4, filter the difficult sample set according to the probability value of each negative sample belonging to the positive sample and the predicted label; S5, calculating a comprehensive score corresponding to each negative sample in the negative sample set according to the diversity score and the difficult sample set; S6, screening the t-th difficult negative sample dataset according to the comprehensive score; S7, merge the t-th difficult negative sample data set and the positive sample set to obtain the t+1-th training set; S8, let the training batch t=t+1, repeat steps S3-S7 to repeatedly optimize the parameters of the model to be trained; until the preset first end condition is met, the training is terminated, and the final tth model to be trained is used as the trained model to be trained.

2. The model training method based on diverse difficult negative sample mining according to claim 1 is characterized by: The diversity assessment network includes an encoder and a discriminator; Step S1 specifically includes: S11, extracting features from negative samples through the encoder to obtain low-dimensional latent features; S12, predicting the probability that the negative sample belongs to the positive sample according to the potential low-dimensional features through the discriminator; S13, mapping the probability that the negative sample belongs to the positive sample into a diversity score.

3. The model training method based on diverse difficult negative sample mining according to claim 2 is characterized by: In step S13, the diversity score D s The calculation formula is: D s ={1-Φ D (I)|I∈D neg }, where D neg represents the negative sample set, I represents the negative sample in the negative sample set, Φ D (I) represents the probability that the negative sample predicted by the discriminator belongs to the positive sample.

4. The model training method based on diverse difficult negative sample mining according to claim 3 is characterized by: The diversity assessment network also includes a generator; The training method of the diversity assessment network comprises the steps of: ST1, obtaining a training set including a plurality of positive samples and negative samples, and inputting the samples in the training set into the encoder one by one; the encoder extracts features from each sample in the training set, and outputs a low-dimensional potential feature corresponding to each sample; ST2, the generator generates a reconstructed input sample according to the low-dimensional potential features extracted by the encoder; ST3, comparing the difference between the reconstructed input sample and the real input sample, and calculating the reconstruction loss function; ST4, the discriminator predicts the probability that each sample in the training set belongs to a positive sample based on the low-dimensional potential features extracted by the encoder; ST5, set the probability label value of the positive sample in the training set to 1, and the probability label value of the negative sample to 0; compare the probability that each sample predicted by the discriminator belongs to the positive sample with the difference between the probability label value, and calculate the discriminator loss function; ST6, setting the adversarial label value of each sample in the training set to 1; comparing the probability that each sample predicted by the discriminator belongs to a positive sample with the difference between the adversarial label value, and calculating the adversarial loss function; ST7, calculating an encoder loss function according to the reconstruction loss function and the adversarial loss function; ST8, iteratively optimizing the parameters of the encoder according to the encoder loss function, and iteratively optimizing the parameters of the discriminator according to the discriminator loss function; until a preset second end condition is met, the training is terminated to obtain a trained diversity evaluation network.

5. The model training method based on diverse difficult negative sample mining according to claim 4 is characterized by: The reconstruction loss function L recon The calculation formula is: Where θ AE represents the parameters of the encoder, I i represents the i-th real input sample, represents the i-th reconstructed input sample, MSE() represents the mean square error, and M represents the total number of samples; The discriminator loss function L dis The calculation formula is: Where θ dis represents the parameters of the discriminator, y i dis represents the probability label value of the i-th sample, represents the probability that the i-th sample predicted by the discriminator belongs to a positive sample, BCE() represents binary cross entropy, and M represents the total number of samples; The adversarial loss function L adv The calculation formula is: Where θ AE represents the parameters of the encoder, y i adv represents the adversarial label value of the i-th sample, represents the probability that the i-th sample predicted by the discriminator belongs to a positive sample, BCE() represents binary cross entropy, and M represents the total number of samples; The encoder loss function L AE The calculation formula is: L AE (θ AE )=L recon +λ·L adv , where L recon represents the reconstruction loss function, L adv represents the adversarial loss function, and λ is a pre-set adversarial coefficient.

6. The model training method based on diverse difficult negative sample mining according to claim 5 is characterized by: Step S4 specifically includes: S41, screening negative samples whose predicted labels are not equal to the true labels to form a first difficult sample set; the first difficult sample set is screened by the following formula: Where H t E represents the first difficult sample set, Ⅱ() represents the indicator function, Represents the predicted label output by the model to be trained, y T Represents the true label of the negative sample; S42, according to the probability value of each negative sample belonging to the positive sample, the probability interval is calculated, and the negative samples whose probability interval is less than the preset interval threshold are screened out to form a second difficult sample set; the second difficult sample set is screened by the following formula: H t M ={Ⅱ(|2p t I -1|<τ)|I∈D neg }, where H t M represents the second difficult sample set, Ⅱ() represents the indicator function, p t I It indicates the probability value that the negative sample belongs to the positive sample, and τ indicates the preset interval threshold; S43, obtaining a combined difficult sample set according to the first difficult sample set and the second difficult sample set; the combined difficult sample set is screened by the following formula: H t =H t E ∨H t M , where H t represents the joint difficult sample set, H t E represents the first difficult sample set, H t M represents the second difficult sample set, and ∨ represents the element-wise OR operation.

7. The model training method based on diverse difficult negative sample mining according to claim 6 is characterized by: Step S5 specifically includes: according to the diversity score D s and difficult sample set H t , calculate the comprehensive score V corresponding to each negative sample in the negative sample set t , the comprehensive score V t Calculated by the following formula: V t =H t ·D s , where V t Indicates the comprehensive score, H t Denotes the difficult sample set, D s Represents the diversity score.

8. The model training method based on diverse difficult negative sample mining according to claim 7 is characterized by: Step S6 specifically includes: screening out the comprehensive score V t The top K negative samples are used as the t-th difficult negative sample dataset D t neg , that is, the t-th difficult negative sample dataset D t neg Filter by the following formula: D t neg =top(V t ,K,D neg ), K is the preset capacity of the difficult negative sample dataset.

9. A model training device based on diverse difficult negative sample mining, characterized in that: include: A sample acquisition module, which is used to acquire positive sample sets and negative sample sets; Diversity evaluation network, which is used to evaluate the diversity score corresponding to each negative sample in the negative sample set; Initialization module, which is used to set the training batch t=0 and the 0th training set equal to the positive sample set; An optimization module, which is used to optimize the parameters of the model to be trained through the tth training set to obtain the tth model to be trained; the negative sample set is input into the tth model to be trained, and the tth model to be trained outputs the probability value of each negative sample in the negative sample set belonging to the positive sample and the corresponding prediction label; The difficulty screening module is used to screen the difficult sample set according to the probability value of each negative sample belonging to the positive sample and the predicted label; A comprehensive scoring module, which is used to calculate a comprehensive score corresponding to each negative sample in the negative sample set according to the diversity score and the difficult sample set; A difficult negative sample screening module, which is used to screen the t-th difficult negative sample data set according to the comprehensive score; A training set merging module is used to merge the t-th difficult negative sample data set and the positive sample set to obtain the t+1-th training set; An iterative module is used to set the training batch t=t+1, call the optimization module, the difficulty screening module, the comprehensive scoring module, the difficult negative sample screening module and the training set merging module, and repeatedly optimize the parameters of the model to be trained; until the preset first end condition is met, the training is terminated, and the final tth model to be trained is used as the trained model to be trained.

10. An electronic device, characterized in that: include: processor; a memory for storing a computer program executed by the processor; Wherein, when the processor executes the computer program, the model training method based on diverse difficult negative sample mining described in any one of claims 1-8 is implemented.

Images