Model training method and device based on diversity difficulty negative sample mining, and equipment

Abstract

The present application relates to a kind of model training method, device and equipment based on diversity difficult negative sample mining.The method includes: the diversity score corresponding to each negative sample in negative sample set is evaluated;Let training batch t=0, let the 0th training set be equal to positive sample set;The parameters of the model to be trained are optimized by the tth training set, and the tth model to be trained is obtained;The negative sample set is input into the tth model to be trained, and the probability value that each negative sample in negative sample set belongs to positive sample and the corresponding predicted label are obtained;According to the probability value that each negative sample belongs to positive sample and predicted label, difficult sample set is screened;According to diversity score and difficult sample set, the comprehensive score corresponding to each negative sample in negative sample set is calculated;According to comprehensive score, the tth difficult negative sample data set is screened;The tth difficult negative sample data set and positive sample set are combined, and the t+1th training set is obtained;Let training batch t=t+1, repeat the parameters of the model to be trained are optimized by the tth training set.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model training method, device and equipment based on diverse difficult negative sample mining. Background Art

[0002] Machine vision is a rapidly developing branch of artificial intelligence. It uses an image acquisition device (such as a camera) to capture images of the target to be inspected. It then uses an AI-based image recognition model to extract and classify features from the image, ultimately identifying relevant information about the target. When applied to industrial product defect detection, machine vision technology can automatically identify defects based on the image of the product being inspected. It boasts accuracy comparable to that of the human eye, extremely high detection speeds, and virtually no wear and tear on the product. Due to these advantages, machine vision technology is currently widely used in the field of industrial product defect detection.

[0003] The core of machine vision technology is an AI-based image recognition model. When a sample image with a true label is input into the image recognition model, the model outputs a predicted label for the sample image. If the true label is T(true), the predicted label is also T(true), and the sample image is called a true positive (TP). If the true label is F(false), but the predicted label is T(true), the sample image is called a false positive (FP). If the true label is T(true), but the predicted label is F(false), the sample image is called a false negative (FN). If the true label is F(false), the predicted label is also F(false), and the sample image is called a true negative (TN). In the field of industrial product defect detection, a label of T(true) indicates the presence of a defect, while a label of F(false) indicates the absence of a defect.

[0004] Hard negative samples are images that are highly similar to positive samples and easily misidentified by the model as false positives (FPs), but are actually negative samples. Hard negative mining involves the specialized mining and processing of hard negative samples during image recognition model training to improve model training and enhance the model's recognition performance. This principle is similar to how humans, when studying, sort out incorrect answers from homework and create a "wrong answer book." By reviewing the "wrong answer book," they can consolidate weak points.

[0005] See also Figure 1 , Figure 1The figure is a flowchart of an existing online difficult example mining method. The OHEM (Online Hard Example Mining) method is an existing difficult negative example mining method, which includes the following steps: S01, copying the image recognition model to be trained into the same first image recognition sub-model and second image recognition sub-model; S02, inputting multiple sample images into the first image recognition sub-model, and the first image recognition sub-model outputs predicted labels; calculating the loss function by comparing the predicted labels with the true labels; screening out several false positive samples with the largest loss function values ​​to form a difficult negative sample data set; S03, inputting the difficult negative sample data set as a training set into the second image recognition sub-model, optimizing the parameters of the second image recognition sub-model; using the optimized second image recognition sub-model as the new first image recognition sub-model, and returning to step S02; S04, repeating steps S02-S03, iteratively optimizing the parameters of the second image recognition sub-model, until the training ends when the preset end condition is met, and using the final second image recognition sub-model as the trained image recognition model.

[0006] Existing OHEM methods have the following flaws: 1. Simply selecting difficult negative samples based on the magnitude of the loss function fails to ensure the diversity and representativeness of the selected difficult negative samples, resulting in limited improvement in model training. 2. Focusing on improving the model's prediction accuracy as a training goal makes the trained image recognition model overly cautious in order to avoid misjudgments. This results in a stricter standard for determining the presence of defects, which can easily lead to missed detections (high numbers of false negative samples). Summary of the Invention

[0007] Based on this, the purpose of the present invention is to provide a model training method, device and electronic device based on diverse difficult negative sample mining, which comprehensively considers the two indicators of diversity and difficulty to screen difficult negative samples, ensures that the screened difficult negative samples are diverse and typical, effectively improves the training effect of the model, and enables the model to achieve a higher recall rate.

[0008] The present invention provides a model training method based on diversity difficult negative sample mining, which comprises the following steps: S0, obtaining a positive sample set and a negative sample set; S1, evaluating the diversity score corresponding to each negative sample in the negative sample set through a diversity evaluation network; S2, setting the training batch t=0, setting the 0th training set equal to the positive sample set; S3, optimizing the parameters of the to-be-trained model through the tth training set to obtain the tth to-be-trained model; inputting the negative sample set into the tth to-be-trained model, and the tth to-be-trained model outputting the probability value of each negative sample in the negative sample set belonging to the positive sample and the corresponding prediction label; S4, according to each The probability value and predicted label of the negative sample belonging to the positive sample are used to screen the difficult sample set; S5, according to the diversity score and the difficult sample set, the comprehensive score corresponding to each negative sample in the negative sample set is calculated; S6, according to the comprehensive score, the t-th difficult negative sample data set is screened; S7, the t-th difficult negative sample data set and the positive sample set are merged to obtain the t+1-th training set; S8, the training batch t=t+1, and steps S3-S7 are repeated to repeatedly optimize the parameters of the model to be trained; until the preset first end condition is met, the training is terminated, and the final t-th model to be trained is used as the trained model to be trained.

[0009] The present invention has the following technical effects: 1. The present invention sets a diversity evaluation network to evaluate the diversity of sample images, and comprehensively considers the two indicators of diversity and difficulty to screen difficult negative samples, ensuring that the screened difficult negative sample data set can cover a variety of error typicalities, improving the quality of the difficult negative sample data set, improving the effect of difficult sample mining, and greatly improving the model performance. 2. Since false negative samples are actually positive samples, and the present invention unconditionally adds the positive sample set when screening the training set, it ensures that all false negative samples are included in the training set of each batch, thereby strengthening the learning of false negative samples by the model to be trained, effectively and efficiently improving the recall rate of the model, and adapting to the needs of the field of industrial product defect detection. 3. The present invention does not need to change the network structure of the model to be trained, nor does it need to obtain more original training data. It only needs to iteratively change the components of the model training set to effectively improve the training effect of the model, and has a high cost performance.

[0010] Furthermore, the diversity assessment network includes an encoder and a discriminator; step S1 specifically includes: S11, using the encoder, extracting features from negative samples to obtain low-dimensional potential features; S12, using the discriminator, predicting the probability that the negative sample belongs to the positive sample based on the potential low-dimensional features; S13, mapping the probability that the negative sample belongs to the positive sample to a diversity score.

[0011] Furthermore, in step S13, the diversity score D s The calculation formula is: s ={1-Φ D(I)|I∈D neg}, where D neg represents the negative sample set, I represents the negative sample in the negative sample set, Φ D (I) represents the probability that the negative sample predicted by the discriminator belongs to the positive sample.

[0012] Furthermore, the diversity evaluation network also includes a generator; the training method for the diversity evaluation network includes the following steps: ST1, obtaining a training set including a number of positive samples and negative samples, and inputting the samples in the training set into the encoder one by one; the encoder extracts features from each sample in the training set and outputs low-dimensional potential features corresponding to each sample; ST2, the generator generates a reconstructed input sample based on the low-dimensional potential features extracted by the encoder; ST3, comparing the difference between the reconstructed input sample and the real input sample, and calculating the reconstruction loss function; ST4, the discriminator predicts the probability that each sample in the training set belongs to a positive sample based on the low-dimensional potential features extracted by the encoder; ST5, making the positive samples in the training set The probability label value of the sample is 1, and the probability label value of the negative sample is 0; compare the probability that each sample predicted by the discriminator belongs to the positive sample with the difference between the probability label value, and calculate the discriminator loss function; ST6, let the adversarial label value of each sample in the training set be 1; compare the probability that each sample predicted by the discriminator belongs to the positive sample with the difference between the adversarial label value, and calculate the adversarial loss function; ST7, calculate the encoder loss function according to the reconstruction loss function and the adversarial loss function; ST8, iteratively optimize the parameters of the encoder according to the encoder loss function, and iteratively optimize the parameters of the discriminator according to the discriminator loss function; until the preset second end condition is met, the training is terminated to obtain a trained diversity evaluation network.

[0013] Furthermore, the reconstruction loss function L recon The calculation formula is: Where θ AE Represents the parameters of the encoder, I i represents the i-th real input sample, represents the i-th reconstructed input sample, MSE() represents the mean square error, and M represents the total number of samples;

[0014] The discriminator loss function L dis The calculation formula is: Where θ dis represents the parameters of the discriminator, y i dis represents the probability label value of the i-th sample, represents the probability that the i-th sample predicted by the discriminator belongs to the positive sample, BCE() represents the binary cross entropy, and M represents the total number of samples;

[0015] The adversarial loss function L adv The calculation formula is: Where θ AE represents the parameters of the encoder, y i adv represents the adversarial label value of the i-th sample, represents the probability that the i-th sample predicted by the discriminator belongs to the positive sample, BCE() represents the binary cross entropy, and M represents the total number of samples;

[0016] The encoder loss function L AE The calculation formula is: L AE (θ AE )=L recon +λ·L adv , where L recon represents the reconstruction loss function, L adv represents the adversarial loss function, and λ is a pre-set adversarial coefficient.

[0017] Furthermore, step S4 specifically includes: S41, screening negative samples whose predicted labels are not equal to the true labels to form a first difficult sample set; the first difficult sample set is screened by the following formula: Where H t E represents the first difficult sample set, Ⅱ() represents the indicator function, Represents the predicted label output by the model to be trained, y T Indicates the true label of the negative sample; S42, according to the probability value of each negative sample belonging to the positive sample, calculate the probability interval, and screen out the negative samples whose probability interval is less than the preset interval threshold to form a second difficult sample set; the second difficult sample set is screened by the following formula: H t M ={Ⅱ(2p t I -1|<τ)I∈D neg}, where H t M represents the second difficult sample set, Ⅱ() represents the indicator function, p t I represents the probability value of a negative sample belonging to a positive sample, and τ represents a preset interval threshold; S43, obtaining a joint difficult sample set based on the first difficult sample set and the second difficult sample set; the joint difficult sample set is screened by the following formula: H t =H t E ∨H t M , where H t Denotes the joint difficult sample set, H tE represents the first difficult sample set, H t M represents the second difficult sample set, and ∨ represents the element-wise OR operation.

[0018] Furthermore, step S5 is specifically as follows: according to the diversity score D s and difficult sample set H t , calculate the comprehensive score V corresponding to each negative sample in the negative sample set t , the comprehensive score V t Calculated by the following formula: V t =H t ·D s , where V t Indicates the comprehensive score, H t Denotes the difficult sample set, D s Represents the diversity score.

[0019] Furthermore, step S6 is specifically as follows: screening out the comprehensive score V t The top K negative samples are used as the t-th difficult negative sample dataset D t neg , that is, the t-th difficult negative sample dataset D t neg Filter by the following formula: D t neg =top(V t ,K,D neg ); K is the preset capacity of the difficult negative sample dataset.

[0020] Based on the same inventive concept, the present application also provides a model training device based on diversity difficult negative sample mining, comprising: a sample acquisition module for acquiring a positive sample set and a negative sample set; a diversity evaluation network for evaluating a diversity score corresponding to each negative sample in the negative sample set; an initialization module for setting a training batch t=0 and setting a 0th training set equal to the positive sample set; an optimization module for optimizing parameters of a to-be-trained model through the tth training set to obtain a tth to-be-trained model; inputting the negative sample set into the tth to-be-trained model, and the tth to-be-trained model outputting a probability value of each negative sample belonging to a positive sample and a corresponding predicted label; a difficulty screening module for screening a difficult sample set according to the probability value of each negative sample belonging to a positive sample and the predicted label; a comprehensive score module for calculating a comprehensive score corresponding to each negative sample in the negative sample set according to the diversity score and the difficult sample set; a difficult negative sample screening module for screening a tth difficult negative sample data set according to the comprehensive score; a training set combining module for combining the tth difficult negative sample data set and the positive sample set to obtain a (t+1)th training set; and an iteration module for setting a training batch t=t+1, calling the optimization module, the difficulty screening module, the comprehensive score module, the difficult negative sample screening module and the training set combining module, and repeatedly optimizing the parameters of the to-be-trained model; until a preset first end condition is met, the training is ended, and a final tth to-be-trained model is taken as a to-be-trained model trained.

[0021] Based on the same inventive concept, the present application also provides an electronic device, comprising: a processor; a memory for storing a computer program executed by the processor; wherein the processor implements any of the above-mentioned model training methods based on diversity difficult negative sample mining when executing the computer program.

[0022] In order to better understand and implement, the present application is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 A flowchart of an existing online difficult sample mining method;

[0024] Figure 2 A module schematic diagram of the model training device based on diversity difficult negative sample mining of the present application;

[0025] Figure 3 A flowchart of the model training method based on diversity difficult negative sample mining of the present application;

[0026] Figure 4 A structure schematic diagram of the diversity evaluation network in an embodiment of the present application;

[0027] Figure 5FIG. 4 is a flow chart of a method for training a diversity assessment network in one embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following embodiments of the technical solution of the present application will be described in detail with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application and are therefore only examples and are not intended to limit the scope of protection of the present application.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned figure descriptions are intended to cover non-exclusive inclusions.

[0030] In the description of the embodiments of this application, the technical terms "first" and "second" are used only to distinguish different objects and should not be understood to indicate or imply relative importance or implicitly specify the quantity, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, the meaning of "plurality" is more than two, unless otherwise clearly and specifically defined.

[0031] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0032] The present invention creatively designs a diversity evaluation network to evaluate the diversity of sample images, and comprehensively considers the two indicators of diversity and difficulty to screen difficult negative samples, ensuring that the screened difficult negative sample dataset can cover a variety of error types, improving the quality of the difficult negative sample dataset, and thus improving the training effect of the model.

[0033] The performance indicators of the image recognition model mainly include accuracy, precision, recall rate and F1 value, among which the calculation formula of recall rate is: It can be seen that the recall rate is equal to the coverage rate of correctly identified positive samples to all positive samples. In the field of industrial product defect detection, the loss caused by the image recognition model misidentifying a negative sample (non-defective) as a positive sample (defective) is much smaller than the loss caused by misidentifying a positive sample (defective) as a negative sample (non-defective). The higher the recall rate of the image recognition model, the lower the probability that the image recognition model misidentifies a positive sample as a negative sample. When screening the training set, the present invention ensures that all false negative samples are included in the training set of each batch, thereby strengthening the learning of false negative samples by the model to be trained, improving the recall rate of the model, and better adapting to the needs of the field of industrial product defect detection.

[0034] See also Figures 2-3 , Figure 2 Schematic diagram of a module of a model training device based on diverse difficult negative sample mining according to the present invention. Figure 3 Schematic diagram of the process of the model training method based on diverse and hard negative sample mining (Diverse and Hard Negatives Mining, DHNM) of the present invention.

[0035] The model training device based on diversity difficult negative sample mining of the present invention is used to train the model to be trained, which includes: a sample acquisition module 0, a diversity evaluation network 1, an initialization module 2, an optimization module 3, a difficulty screening module 4, a comprehensive scoring module 5, a difficult negative sample screening module 6, a training set merging module 7 and an iteration module 8.

[0036] The sample acquisition module 0 is used to execute step S0: obtain a positive sample set D pos and negative sample set D neg .

[0037] The diversity evaluation network 1 is used to perform step S1: evaluate the negative sample set D neg The diversity score D corresponding to each negative sample in s .

[0038] In this embodiment, since the model to be trained is an image recognition model for defect detection, the positive sample set D pos The negative sample set D includes several positive sample images with the real label T (defective, value = 1). neg The training model includes several negative sample images whose true label is F (no defect, value = 0). In other embodiments, the training model can be other classification recognition models, and the present invention does not limit the specific use of the training model.

[0039] The diversity evaluation network 1 is used to evaluate the diversity score corresponding to each negative sample in the negative sample set. The higher the diversity score of the negative sample, the greater the difference between the negative sample and the positive sample, which helps the model to be trained to master the difference between the positive sample and the negative sample and identify a wider range of defect-free patterns.

[0040] In one embodiment, the diversity evaluation network 1 can include a feature extraction unit, a cosine distance calculation unit, and a mapping unit. The feature extraction unit (such as VGGNet or an autoencoder) performs feature extraction on each sample image to obtain a feature vector. The cosine distance calculation unit is used to calculate the cosine distance between the feature vector corresponding to each negative sample and the feature vector corresponding to each positive sample, thereby measuring the difference between each negative sample and the positive sample. The mapping unit maps the cosine distance to a diversity score, so that the greater the difference between the negative sample and the positive sample, the higher the diversity score. This diversity evaluation method based on cosine distance needs to calculate the cosine distance between the feature vector corresponding to each negative sample and the feature vector corresponding to each positive sample, which has a large amount of calculation and takes a long time. Moreover, the cosine distance only considers the image content and lacks information related to a specific task.

[0041] Please refer to Figure 4 , Figure 4 is a structural schematic diagram of the diversity evaluation network in one embodiment of the present application. In this embodiment, the diversity evaluation network 1 includes an encoder 11, a discriminator 12, a mapping module 13, and a generator 14. The encoder 11 includes a plurality of ConvBlock layers and a fully connected layer cascaded in sequence; each ConvBlock layer includes a convolution layer, a BN layer, and a ReLU layer cascaded in sequence. The discriminator 12 includes two FCBlock layers, a fully connected layer, and a Sigmoid layer cascaded in sequence; each FCBlock layer includes a fully connected layer, a BN layer, and a ReLU layer cascaded in sequence. The generator 14 includes a fully connected layer and a plurality of TConvBlock layers cascaded in sequence; each TConvBlock layer includes a transposed convolution layer, a BN layer, and a ReLU layer cascaded in sequence.

[0042] The encoder 11 is configured to perform step S11 of performing feature extraction on an input sample I to obtain a low-dimensional latent feature z. The discriminator 12 is configured to perform step S12 of predicting a probability Φ D (I) of the input sample I belonging to a positive sample according to the latent low-dimensional feature z. The mapping module 13 is configured to perform step S13 of mapping the probability Φ D (I) of the input sample I belonging to a positive sample to a diversity score D s, thus ensuring that the greater the difference between negative samples and positive samples, the greater the diversity score D s The higher the value, the more diversity the mapping module calculates. s The formula is:

[0043] D s ={1-Φ D (I)|I∈D neg}, where D neg represents the negative sample set, I represents the negative sample in the negative sample set, Φ D (I) represents the probability that the negative sample predicted by the discriminator belongs to the positive sample.

[0044] The generator 14 is used to generate a reconstructed input sample according to the low-dimensional potential feature z The reconstructed input sample The true label of is the same as the true label of the input sample I. The reconstructed input sample generated by the generator 14 Used to perform adversarial training on the encoder 11 and the discriminator 12.

[0045] See also Figure 5 , Figure 5 FIG. 1 is a flow chart of a method for training a diversity assessment network in accordance with an embodiment of the present invention. The method for performing adversarial training on the encoder 11 and the discriminator 12 (i.e., the method for training the diversity assessment network) includes:

[0046] ST1, obtain a training set including several positive samples and negative samples, and input the samples in the training set into the encoder one by one; the encoder extracts features from each sample in the training set and outputs a low-dimensional potential feature z corresponding to each sample.

[0047] ST2, the generator generates a reconstructed input sample based on the low-dimensional potential feature z extracted by the encoder

[0048] ST3, compare the reconstructed input samples The difference from the real input sample I is used to calculate the reconstruction loss function L recon , the calculation formula of the reconstruction loss function is:

[0049] Where θ AE Represents the parameters of the encoder, I i represents the i-th real input sample, represents the i-th reconstructed input sample, MSE() represents the mean square error, and M represents the total number of samples.

[0050] ST4: The discriminator predicts the probability that each sample in the training set is a positive sample based on the low-dimensional potential features extracted by the encoder.

[0051] ST5: Let the probability label value of the positive sample in the training set be 1 and the probability label value of the negative sample be 0; compare the probability of each sample predicted by the discriminator belonging to the positive sample with the difference between the probability label value and calculate the discriminator loss function L dis , the discriminator loss function L dis The calculation formula is:

[0052] Where θ dis represents the parameters of the discriminator, y i dis represents the probability label value of the i-th sample (positive sample is 1, negative sample is 0), represents the probability that the i-th sample predicted by the discriminator belongs to the positive sample, BCE() represents the binary cross entropy, and M represents the total number of samples.

[0053] ST6, let the adversarial label value of each sample in the training set be 1; compare the probability of each sample predicted by the discriminator as a positive sample with the difference between the adversarial label value, and calculate the adversarial loss function L adv , the adversarial loss function L adv The calculation formula is:

[0054] Where θ AE represents the parameters of the encoder, y i adv represents the adversarial label value of the i-th sample, represents the probability that the i-th sample predicted by the discriminator belongs to the positive sample, BCE() represents the binary cross entropy, and M represents the total number of samples.

[0055] ST7, based on the reconstruction loss function L recon And the adversarial loss function L adv , calculate the encoder loss function L AE , the encoder loss function L AE The calculation formula is:

[0056] L AE (θ AE )=L recon +λ·L adv , where L recon represents the reconstruction loss function, L advDenotes the adversarial loss function, and λ is a pre-set adversarial coefficient (hyperparameter). In this embodiment, in order to prevent the low-dimensional latent features z extracted by the encoder from being unable to effectively represent the input image, the adversarial coefficient λ is set to 0 in the first several training batches.

[0057] ST8, according to the encoder loss function L AE Iteratively optimize the parameters of the encoder according to the discriminator loss function L dis The parameters of the discriminator are iteratively optimized until a preset second termination condition is met, whereupon the training terminates to obtain a trained diversity assessment network (i.e., a trained encoder and discriminator). In this embodiment, the preset second termination condition is: the number of iterations of the encoder and discriminator parameters reaches a preset second threshold or the encoder and discriminator parameters converge.

[0058] The initialization module 2 is used to execute step S2: record the training batch t=0, set the 0th training set D0 equal to the positive sample set D pos .

[0059] The optimization module 3 is used to execute step S3: the t-th training set D t Input it into the model to be trained, optimize the parameters of the model to be trained, and obtain the tth model to be trained Φ t T ; Set the negative sample set D neg Input to the tth to-be-trained model Φ t T In the example, the tth model to be trained Φ t T Output the probability value of each negative sample in the negative sample set belonging to the positive sample and the corresponding predicted label.

[0060] Specifically, the probability value of the negative sample output by the training model belongs to the positive sample When this probability value When , the corresponding prediction label is T (value = 1), indicating that the model to be trained determines that the negative sample (no defect) belongs to the positive sample (defective); when this probability value When , the corresponding prediction label is F (value = 0), indicating that the to-be-trained model determines that the negative sample (no defect) does not belong to the positive sample (with defect).

[0061] The difficulty screening module 4 is used to perform step S4: screening the difficult sample set H according to the probability value of each negative sample belonging to the positive sample and the predicted label. t .

[0062] Step S4 includes the following steps:

[0063] S41, screening negative samples whose predicted labels are not equal to the true labels to form a first difficult sample set; the first difficult sample set is screened by the following formula:

[0064] Where H t E represents the first difficult sample set, Ⅱ() represents the indicator function, Represents the predicted label output by the model to be trained, y T Denotes the true label of the negative sample (value = 0). The first difficult sample set H t E In , negative samples that are screened as difficult samples are marked as 1, and other simple samples are marked as 0.

[0065] S42, calculating the probability interval based on the probability value of each negative sample belonging to the positive sample, and screening out negative samples with a probability interval less than a preset interval threshold to form a second difficult sample set; the second difficult sample set is screened by the following formula:

[0066] H t M ={Ⅱ(|2p t I -1|<τ)|I∈D neg}, where H t M represents the second difficult sample set, Ⅱ() represents the indicator function, p t I represents the probability value of the negative sample belonging to the positive sample, and τ represents the preset interval threshold. The second difficult sample set H t M In , negative samples that are screened as difficult samples are marked as 1, and other simple samples are marked as 0.

[0067] The probability interval refers to the probability value p that the trained model predicts that a sample belongs to a positive sample. t I , and the difference between the probability value of predicting that the sample belongs to the negative sample, which is equal to |2p t I A smaller probability interval indicates that the trained model has less confidence in the correctness of the prediction and the prediction is more difficult.

[0068] S43, obtaining a combined difficult sample set based on the first difficult sample set and the second difficult sample set; the combined difficult sample set is filtered by the following formula:

[0069] H t =H t E ∨H t M , where Ht Denotes the joint difficult sample set, H t E represents the first difficult sample set, H t M Denotes the second difficult sample set, ∨ denotes the element-by-element OR operation. Joint difficult sample set H t In , negative samples that are screened as difficult samples are marked as 1, and other simple samples are marked as 0.

[0070] The comprehensive scoring module 5 is used to execute step S5: according to the diversity score D s and difficult sample set H t , calculate the comprehensive score V corresponding to each negative sample in the negative sample set t , the comprehensive score V t Calculated by the following formula:

[0071] V t =H t ·D s , where V t Indicates the comprehensive score, H t Denotes the difficult sample set, D s Represents the diversity score.

[0072] The difficult negative sample screening module 6 is used to perform step S6: screening the comprehensive score V t The top K negative samples are used as the t-th difficult negative sample dataset D t neg , that is, the t-th difficult negative sample dataset D t neg Filter by the following formula: D t neg =top(V t ,K,D neg ).

[0073] The training set merging module 7 is used to execute step S7: let the t-th difficult negative sample dataset D t neg and positive sample set D pos Merge to get the t+1th training set D t+1 .

[0074] The iterative module 8 is used to execute step S8: let the training batch t=t+1, call the optimization module, difficulty screening module, comprehensive scoring module, difficult negative sample screening module and training set merging module to repeat steps S3-S8 to repeatedly optimize the parameters of the model to be trained; until the preset first end condition is met, the training is terminated, and the final t-th model to be trained Φ is t TIn this embodiment, the first preset end condition is that the training batch t reaches a first preset number threshold or the parameters of the model to be trained converge.

[0075] The present invention has the following technical effects: 1. The present invention creatively designs a diversity evaluation network to evaluate the diversity of sample images, and comprehensively considers the two indicators of diversity and difficulty to screen difficult negative samples, ensuring that the screened difficult negative sample data set can cover a variety of error typicalities, improving the quality of the difficult negative sample data set, improving the effect of difficult sample mining, and greatly improving the model performance. 2. Since false negative samples (FN) are actually positive samples, and the present invention unconditionally adds the positive sample set when screening the training set, it ensures that all false negative samples are included in the training set of each batch, thereby strengthening the learning of false negative samples by the model to be trained, effectively and efficiently improving the recall rate of the model, and adapting to the needs of the field of industrial product defect detection. 3. The present invention does not need to change the network structure of the model to be trained, nor does it need to obtain more original training data. It only needs to iteratively change the components of the model training set to effectively improve the training effect of the model, and has a high cost performance. Experimental results show that the model training method based on diversity difficult negative sample mining proposed by the present invention is effective in P 0.99 The performance of YOLO X and YOLO v4 has been improved by more than 13% in terms of indicators. At the same time, similar performance improvements have been observed on the public KolektorSDD2 dataset, specifically P 0.99 The indicator grew by about 5%.

[0076] Based on the same inventive concept, the present application also provides an electronic device, which can be a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). The device includes one or more processors and a memory, wherein the processor is used to execute a program to implement the model training method based on diverse difficult negative sample mining according to an embodiment of the present invention; and the memory is used to store a computer program executable by the processor.

[0077] Based on the same inventive concept, the present application also provides a computer-readable storage medium, corresponding to the embodiment of the aforementioned model training method based on diverse difficult negative sample mining, wherein the computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the model training method based on diverse difficult negative sample mining recorded in any of the aforementioned embodiments.

[0078] The present application may take the form of a computer program product implemented on one or more storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. The information can be computer-readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0079] The above-described embodiments merely represent several implementations of the present invention. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, and the present invention is intended to encompass such modifications and variations.

Claims

1. A model training method based on diverse difficult negative sample mining, characterized by: Including steps: S0, obtaining a positive sample set and a negative sample set; the data in the positive sample set and the negative sample set are sample image data; S1, through a diversity evaluation network, evaluates the diversity score corresponding to each negative sample in the negative sample set; S2, let training batch t = 0, and let the 0th training set be equal to the positive sample set; S3, optimizing the parameters of the model to be trained using the tth training set to obtain the tth model to be trained; inputting the negative sample set into the tth model to be trained, and the tth model to be trained outputs the probability value of each negative sample in the negative sample set belonging to the positive sample and the corresponding prediction label; S4, based on the probability value of each negative sample belonging to the positive sample and the predicted label, filter the difficult sample set; S5, calculating a comprehensive score corresponding to each negative sample in the negative sample set based on the diversity score and the difficult sample set; S6, screening the t-th difficult negative sample dataset according to the comprehensive score; S7, merge the t-th difficult negative sample dataset and the positive sample set to obtain the t+1-th training set; S8, let training batch t=t+1, repeat steps S3-S7 to repeatedly optimize the parameters of the model to be trained; until the preset first end condition is met, end the training, and use the final tth model to be trained as the trained model to be trained.

2. The model training method based on diverse difficult negative sample mining according to claim 1, characterized in that: The diversity assessment network includes an encoder and a discriminator; Step S1 specifically includes: S11, extracting features from negative samples through the encoder to obtain low-dimensional latent features; S12, predicting, by the discriminator, the probability that the negative sample belongs to a positive sample based on the potential low-dimensional features; S13, mapping the probability that the negative sample belongs to the positive sample into a diversity score.

3. The model training method based on diverse difficult negative sample mining according to claim 2, characterized in that: In step S13, the diversity score D s The calculation formula is: D s ={1-Φ D (I)|I∈D neg }, where D neg represents the negative sample set, I represents the negative sample in the negative sample set, Φ D (I) represents the probability that the negative sample predicted by the discriminator belongs to the positive sample.

4. The model training method based on diverse difficult negative sample mining according to claim 3, characterized in that: The diversity assessment network also includes a generator; The training method for the diversity assessment network comprises the steps of: ST1, obtain a training set including several positive samples and negative samples, and input the samples in the training set into the encoder one by one; the encoder extracts features from each sample in the training set and outputs low-dimensional potential features corresponding to each sample; ST2, the generator generates a reconstructed input sample based on the low-dimensional potential features extracted by the encoder; ST3, comparing the difference between the reconstructed input sample and the real input sample, and calculating the reconstruction loss function; ST4, the discriminator predicts the probability that each sample in the training set is a positive sample based on the low-dimensional potential features extracted by the encoder; ST5, let the probability label value of the positive sample in the training set be 1, and the probability label value of the negative sample be 0; compare the probability of each sample belonging to the positive sample predicted by the discriminator with the difference between the probability label value, and calculate the discriminator loss function; ST6, setting the adversarial label value of each sample in the training set to 1; comparing the probability of each sample predicted by the discriminator as a positive sample with the difference between the adversarial label value, and calculating the adversarial loss function; ST7, calculating an encoder loss function based on the reconstruction loss function and the adversarial loss function; ST8, iteratively optimize the parameters of the encoder according to the encoder loss function, and iteratively optimize the parameters of the discriminator according to the discriminator loss function; until a preset second end condition is met, end the training and obtain a trained diversity evaluation network.

5. The model training method based on diverse difficult negative sample mining according to claim 4, characterized in that: The reconstruction loss function L recon The calculation formula is: Where θ AE Represents the parameters of the encoder, I i represents the i-th real input sample, represents the i-th reconstructed input sample, MSE() represents the mean square error, and M represents the total number of samples; The discriminator loss function L dis The calculation formula is: Where θ dis represents the parameters of the discriminator, y i dis represents the probability label value of the i-th sample, represents the probability that the i-th sample predicted by the discriminator belongs to the positive sample, BCE() represents the binary cross entropy, and M represents the total number of samples; The adversarial loss function L adv The calculation formula is: Where θ AE represents the parameters of the encoder, y i adv represents the adversarial label value of the i-th sample, represents the probability that the i-th sample predicted by the discriminator belongs to the positive sample, BCE() represents the binary cross entropy, and M represents the total number of samples; The encoder loss function L AE The calculation formula is: L AE (θ AE )=L recon +λ·L adv , where L recon represents the reconstruction loss function, L adv represents the adversarial loss function, and λ is a pre-set adversarial coefficient.

6. The model training method based on diverse difficult negative sample mining according to claim 5, characterized in that: Step S4 specifically includes: S41, screening negative samples whose predicted labels are not equal to the true labels to form a first difficult sample set; the first difficult sample set is screened by the following formula: Where H t E represents the first difficult sample set, Ⅱ() represents the indicator function, Represents the predicted label output by the model to be trained, y T Represents the true label of the negative sample; S42, calculating the probability interval based on the probability value of each negative sample belonging to the positive sample, and screening out negative samples with a probability interval less than a preset interval threshold to form a second difficult sample set; the second difficult sample set is screened by the following formula: H t M ={Ⅱ(|2p t I -1|<τ)|I∈D neg }, where H t M represents the second difficult sample set, Ⅱ() represents the indicator function, p t I It represents the probability value of the negative sample belonging to the positive sample, and τ represents the preset interval threshold; S43, obtaining a combined difficult sample set based on the first difficult sample set and the second difficult sample set; the combined difficult sample set is filtered by the following formula: H t =H t E ∨H t M , where H t Denotes the joint difficult sample set, H t E represents the first difficult sample set, H t M represents the second difficult sample set, and ∨ represents the element-wise OR operation.

7. The model training method based on diverse difficult negative sample mining according to claim 6, characterized in that: Step S5 is specifically as follows: according to the diversity score D s and difficult sample set H t , calculate the comprehensive score V corresponding to each negative sample in the negative sample set t , the comprehensive score V t Calculated by the following formula: V t =H t ·D s , where V t Indicates the comprehensive score, H t Denotes the difficult sample set, D s Represents the diversity score.

8. The model training method based on diverse difficult negative sample mining according to claim 7, characterized in that: Step S6 is specifically as follows: filter out the comprehensive score V t The top K negative samples are used as the t-th difficult negative sample dataset D t neg , that is, the t-th difficult negative sample dataset D t neg Filter by the following formula: D t neg =top(V t ,K,D neg ); K is the preset capacity of the difficult negative sample dataset.

9. A model training device based on diverse difficult negative sample mining, characterized in that: include: A sample acquisition module, which is used to acquire a positive sample set and a negative sample set; the data in the positive sample set and the negative sample set are sample image data; Diversity evaluation network, which is used to evaluate the diversity score corresponding to each negative sample in the negative sample set; Initialization module, which is used to set training batch t=0 and set the 0th training set equal to the positive sample set; An optimization module is configured to optimize the parameters of the model to be trained using the tth training set to obtain the tth model to be trained; the negative sample set is input into the tth model to be trained, and the tth model to be trained outputs a probability value of each negative sample in the negative sample set being a positive sample and a corresponding prediction label; The difficulty screening module is used to screen the difficult sample set based on the probability value of each negative sample belonging to the positive sample and the predicted label; A comprehensive scoring module, configured to calculate a comprehensive score corresponding to each negative sample in the negative sample set based on the diversity score and the difficult sample set; a difficult negative sample screening module, configured to screen the t-th difficult negative sample dataset according to the comprehensive score; The training set merging module is used to merge the t-th difficult negative sample dataset and the positive sample dataset to obtain the t+1-th training set; The iterative module is used to set the training batch t=t+1, call the optimization module, the difficulty screening module, the comprehensive scoring module, the difficult negative sample screening module and the training set merging module, and repeatedly optimize the parameters of the model to be trained; until the preset first end condition is met, the training is ended, and the final tth model to be trained is used as the trained model to be trained.

10. An electronic device, characterized in that: include: processor; a memory for storing a computer program executed by the processor; Wherein, when the processor executes the computer program, the model training method based on diverse difficult negative sample mining according to any one of claims 1 to 8 is implemented.

Images