Image sample generation method and device, computer device and storage medium

Abstract

The application relates to an image sample generation method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: based on a network model, the features of image samples in a defect image sample set are extracted to form an image feature vector. Then, according to a product defect judgment rule, a semantic-based judgment criterion is formulated, a semantic description feature vector of the product defect is formed, and the image feature vector is fused. The feature data is trained through a discriminator and a generator to realize the generation of an image sample of a target defect type. Based on the extraction and fusion of the semantic description features of the defect types, the defect feature recognition accuracy of the defect image is improved. Through the fusion of the semantic description features, the training of the generator and the discriminator model, and the expansion of the defect image samples of the small sample product, the efficiency and accuracy of real-time defect recognition can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image sample generation method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] The global 3C industry (referring to the information and home appliance industry that integrates the three major technologies of computers, communications, and consumer electronics) is currently developing rapidly. Taking smartphones as an example, the current annual global smartphone production has reached 1.4 billion units. Printed Circuit Boards (PCBs), as the working carriers of electronic components in the 3C industry, are crucial to the core performance and quality of products.

[0003] There are two main types of methods for PCB defect detection: traditional manual PCB defect detection and intelligent PCB defect detection based on deep learning. Due to the mass production of products, traditional manual inspection inevitably suffers from low efficiency. Furthermore, because PCB defects cover a wide range of types, including burrs and rodent bites, and the circuit structures themselves vary greatly, false positives and false negatives are common. While deep learning-based visual inspection methods possess powerful feature learning capabilities, their training process in real-world production scenarios requires accumulating a large number of image samples to ensure sufficient data for learning each type of defect feature. In practical applications, it is difficult to meet this ideal condition of sufficient samples, which affects the learning performance of the deep learning model and consequently the accuracy of the defect detection results. Summary of the Invention

[0004] Therefore, it is necessary to provide an efficient and accurate method, apparatus, computer device, computer-readable storage medium, and computer program product for generating image samples to address the aforementioned technical problems.

[0005] Firstly, this application provides an image sample generation method. The method includes:

[0006] Obtain a defect image sample set, and determine the target defect type and at least one corresponding target defect image sample based on the defect type of each image sample in the defect image sample set;

[0007] Obtain a set of random image samples generated by the generator to be trained, and determine the image feature vector of each image sample based on the random image sample set and the image data of each image sample in at least one target defect image sample.

[0008] The image feature vector of each image sample in the random image sample set and at least one target defect image sample is input into the first discriminator to obtain the first loss function value of the first discriminator. Based on the first loss function value, the generator to be trained and the first discriminator are trained for the first time to obtain the random image sample set that meets the preset conditions, the generator to be trained after the first training and the first discriminator.

[0009] Obtain the semantic description feature vector of each image sample in the random image sample set and the defect image sample set that meet the preset conditions. Input the semantic description feature vector into the second discriminator to obtain the second loss function value of the second discriminator. Based on the first loss function value and the second loss function value, perform a second training on the generator to be trained, the first discriminator, and the second discriminator after the first training to obtain the generator to be trained, the first discriminator, and the second discriminator after the second training.

[0010] Based on the generator to be trained after the second training, the first discriminator, and the second discriminator, random image samples with the defect type of the target defect type are generated.

[0011] In one embodiment, determining the target defect type and at least one corresponding target defect image sample based on the defect type of each image sample in the defect image sample set includes:

[0012] Based on the defect type of each image sample in the defect image sample set, count the number of image samples of each defect type in the defect image sample set.

[0013] The defect type with the smallest number of image samples and not less than 1 is selected as the target defect type.

[0014] In one embodiment, obtaining a set of random image samples generated by the generator to be trained includes:

[0015] Random sampling based on Gaussian distribution yields multiple noise vectors;

[0016] Based on multiple noise vectors, a pixel matrix of a preset size is generated, and each pixel matrix is ​​used as a random image sample.

[0017] In one embodiment, the image data includes image size and pixel values; accordingly, based on the image data of each image sample in the random image sample set and at least one target defect image sample, an image feature vector for each image sample is determined, including:

[0018] For any image sample in the random image sample set and at least one target defect image sample, determine whether the image size of any image sample is a preset size. If the image size of any image sample is not a preset size, perform size compensation or image cropping on any image sample to obtain any preprocessed image sample.

[0019] The pixel values ​​of any image sample after preprocessing are normalized, and the features of any image sample after pixel normalization are extracted to obtain the image feature vector of any image sample.

[0020] In one embodiment, the image feature vector of each image sample from a random image sample set and at least one target defect image sample is input into a first discriminator to obtain a first loss function of the first discriminator, including:

[0021] The image feature vectors of all image samples in the random image sample set and at least one target defect image sample are input into the first discriminator to obtain the true or false result of each image sample in the random image sample set;

[0022] The first loss function value is calculated based on the true / false results of all image samples in the random image sample set and the manual annotation results of at least one target defect image sample.

[0023] In one embodiment, the generator and the first discriminator are trained for the first time based on the first loss function value, resulting in a random image sample set that meets preset conditions, the generator to be trained after the first training, and the first discriminator, including:

[0024] If the first loss function value is not less than the preset threshold, adjust the optimization parameters of the generator to be trained and the first discriminator, return to the step of obtaining the random image sample set generated by the generator to be trained, continue to train the generator to be trained and the first discriminator for the first time, and count the number of loops.

[0025] If the first loss function is less than a preset threshold or the number of iterations is greater than a preset number, a random image sample set that meets the preset conditions, a generator to be trained after the first training, and a first discriminator are obtained.

[0026] In one embodiment, the second discriminator includes multiple semantic feature sub-discriminators. These multiple semantic feature sub-discriminators are used to determine the semantic feature type of the image sample. The semantic description feature vector is input into the second discriminator to obtain the second loss function value of the second discriminator, including:

[0027] The semantic description feature vectors are input into each semantic feature sub-discriminator to obtain each discrimination result;

[0028] The second loss function value of the second discriminator is calculated based on the number of target discrimination results in each discrimination result.

[0029] In one embodiment, based on the first loss function value and the second loss function value, the generator to be trained after the first training, the first discriminator, and the second discriminator are trained a second time to obtain the generator to be trained after the second training, the first discriminator, and the second discriminator, including:

[0030] The fusion loss function value is determined based on the first loss function value and the second loss function value.

[0031] Based on the fusion loss function value, the generator to be trained, the first discriminator, and the second discriminator after the first training are trained a second time to obtain the generator to be trained, the first discriminator, and the second discriminator after the second training.

[0032] Secondly, this application also provides an image sample generation apparatus. The apparatus includes:

[0033] The sample acquisition module is used to acquire a defect image sample set, and determine the target defect type and at least one corresponding target defect image sample based on the defect type of each image sample in the defect image sample set.

[0034] The feature extraction module is used to obtain a set of random image samples generated by the generator to be trained, and to determine the image feature vector of each image sample based on the random image sample set and the image data of each image sample in at least one target defect image sample.

[0035] The first discrimination module is used to input the image feature vector of each image sample in the random image sample set and at least one target defect image sample into the first discriminator to obtain the first loss function value of the first discriminator, and perform the first training on the generator to be trained and the first discriminator based on the first loss function value to obtain the random image sample set that meets the preset conditions, the generator to be trained after the first training and the first discriminator.

[0036] The second discrimination module is used to obtain the semantic description feature vector of each image sample in the random image sample set and the defect image sample set that meet the preset conditions, input the semantic description feature vector into the second discriminator, obtain the second loss function value of the second discriminator, and perform a second training on the generator to be trained, the first discriminator and the second discriminator after the first training based on the first loss function value and the second loss function value, to obtain the generator to be trained, the first discriminator and the second discriminator after the second training.

[0037] The sample generation module is used to generate random image samples with the defect type of the target defect type based on the generator to be trained after the second training, the first discriminator, and the second discriminator.

[0038] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0039] Obtain a defect image sample set, and determine the target defect type and at least one corresponding target defect image sample based on the defect type of each image sample in the defect image sample set;

[0040] Obtain a set of random image samples generated by the generator to be trained, and determine the image feature vector of each image sample based on the random image sample set and the image data of each image sample in at least one target defect image sample.

[0041] The image feature vector of each image sample in the random image sample set and at least one target defect image sample is input into the first discriminator to obtain the first loss function value of the first discriminator. Based on the first loss function value, the generator to be trained and the first discriminator are trained for the first time to obtain the random image sample set that meets the preset conditions, the generator to be trained after the first training and the first discriminator.

[0042] Obtain the semantic description feature vector of each image sample in the random image sample set and the defect image sample set that meet the preset conditions. Input the semantic description feature vector into the second discriminator to obtain the second loss function value of the second discriminator. Based on the first loss function value and the second loss function value, perform a second training on the generator to be trained, the first discriminator, and the second discriminator after the first training to obtain the generator to be trained, the first discriminator, and the second discriminator after the second training.

[0043] Based on the generator to be trained after the second training, the first discriminator, and the second discriminator, random image samples with the defect type of the target defect type are generated.

[0044] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0045] Obtain a defect image sample set, and determine the target defect type and at least one corresponding target defect image sample based on the defect type of each image sample in the defect image sample set;

[0046] Obtain a set of random image samples generated by the generator to be trained, and determine the image feature vector of each image sample based on the random image sample set and the image data of each image sample in at least one target defect image sample.

[0047] The image feature vector of each image sample in the random image sample set and at least one target defect image sample is input into the first discriminator to obtain the first loss function value of the first discriminator. Based on the first loss function value, the generator to be trained and the first discriminator are trained for the first time to obtain the random image sample set that meets the preset conditions, the generator to be trained after the first training and the first discriminator.

[0048] Obtain the semantic description feature vector of each image sample in the random image sample set and the defect image sample set that meet the preset conditions. Input the semantic description feature vector into the second discriminator to obtain the second loss function value of the second discriminator. Based on the first loss function value and the second loss function value, perform a second training on the generator to be trained, the first discriminator, and the second discriminator after the first training to obtain the generator to be trained, the first discriminator, and the second discriminator after the second training.

[0049] Based on the generator to be trained after the second training, the first discriminator, and the second discriminator, random image samples with the defect type of the target defect type are generated.

[0050] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0051] Obtain a defect image sample set, and determine the target defect type and at least one corresponding target defect image sample based on the defect type of each image sample in the defect image sample set;

[0052] Obtain a set of random image samples generated by the generator to be trained, and determine the image feature vector of each image sample based on the random image sample set and the image data of each image sample in at least one target defect image sample.

[0053] The image feature vector of each image sample in the random image sample set and at least one target defect image sample is input into the first discriminator to obtain the first loss function value of the first discriminator. Based on the first loss function value, the generator to be trained and the first discriminator are trained for the first time to obtain the random image sample set that meets the preset conditions, the generator to be trained after the first training and the first discriminator.

[0054] Obtain the semantic description feature vector of each image sample in the random image sample set and the defect image sample set that meet the preset conditions. Input the semantic description feature vector into the second discriminator to obtain the second loss function value of the second discriminator. Based on the first loss function value and the second loss function value, perform a second training on the generator to be trained, the first discriminator, and the second discriminator after the first training to obtain the generator to be trained, the first discriminator, and the second discriminator after the second training.

[0055] Based on the generator to be trained after the second training, the first discriminator, and the second discriminator, random image samples with the defect type of the target defect type are generated.

[0056] The aforementioned image sample generation method, apparatus, computer equipment, storage medium, and computer program product acquire a defect image sample set; determine the target defect type and at least one corresponding target defect image sample based on the defect type of each image sample in the defect image sample set; acquire a random image sample set generated by the generator to be trained; determine the image feature vector of each image sample based on the image data of each image sample in the random image sample set and at least one target defect image sample; input the image feature vector of each image sample in the random image sample set and at least one target defect image sample into a first discriminator to obtain the first loss function value of the first discriminator; and perform a first loss function test on the generator to be trained and the first discriminator based on the first loss function value. The training process involves obtaining a random image sample set that meets preset conditions, a generator to be trained after the first training, and a first discriminator. Semantic description feature vectors are obtained for each image sample in the random image sample set and the defect image sample set that meet preset conditions. These semantic description feature vectors are input into a second discriminator to obtain its second loss function value. Based on the first and second loss function values, the generator to be trained after the first training, the first discriminator, and the second discriminator are trained a second time to obtain the generator to be trained after the second training, the first discriminator, and the second discriminator. Based on the generator to be trained after the second training, the first discriminator, and the second discriminator, random image samples with the target defect type are generated. The extraction and fusion of semantic description features based on defect type improves the accuracy of defect feature recognition in defect images. By fusing semantic description features to train the generator and discriminator models, the defect image samples of small-sample products are expanded, improving the efficiency and accuracy of real-time defect recognition. Attached Figure Description

[0057] Figure 1 This is an application environment diagram of an image sample generation method in one embodiment;

[0058] Figure 2 This is a flowchart illustrating an image sample generation method in one embodiment;

[0059] Figure 3 This is a schematic diagram of the generator fully connected network model structure in one embodiment;

[0060] Figure 4 This is a schematic diagram of the generator convolutional network model structure in one embodiment;

[0061] Figure 5 This is a schematic diagram of the network model structure of the first discriminator in one embodiment;

[0062] Figure 6 This is a flowchart illustrating the image sample generation method in another embodiment;

[0063] Figure 7 This is a flowchart illustrating the image sample generation method in yet another embodiment;

[0064] Figure 8 This is a flowchart illustrating a semantic rule-based defect image generation method in one embodiment;

[0065] Figure 9 This is a structural block diagram of an image sample generation device in one embodiment;

[0066] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0068] The image sample generation method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0069] In one embodiment, such as Figure 2 As shown, an image sample generation method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0070] Step 202: Obtain a defect image sample set, and determine the target defect type and at least one corresponding target defect image sample based on the defect type of each image sample in the defect image sample set.

[0071] The defect image sample set is a collection of defect images obtained through defect identification and judgment of products. Each defect image serves as a training sample for the defect detection model. For products in mass production that require defect detection, the defect image sample set can be obtained from the quality inspection results of the initial production batches. Specifically, each defect image sample in the defect image sample set includes an image of the defective part and defect information obtained through image recognition and other processing, such as the defect type. The defect type describes the appearance and cause of the defect in the product; for example, defect types include burrs and rodent bites.

[0072] Defect type is a crucial indicator in product defect detection; therefore, the defect detection model needs to learn the features of defect image samples for each defect type. It's important to note that the defect type of each image sample in the defect image sample set can be determined manually or through preliminary detection by an existing defect detection model. The target defect type is the only one among all defect types that needs to be learned. This target defect type can be manually set or determined based on the defect types of each image sample in the defect image sample set. Furthermore, it's crucial to emphasize that in a single optimization training of the generator, only one target defect type is used. However, multiple target defect types can be identified through the defect image sample set, allowing for the optimization of multiple corresponding generators.

[0073] Understandably, the purpose of determining the target defect type is twofold: first, to identify image samples in the defect image sample set that belong to the target defect type; and second, to train the generator model using these target defect image samples. Therefore, the number of target defect image samples with the target defect type must be at least one, and theoretically, this number cannot be too small.

[0074] Step 204: Obtain a random image sample set generated by the generator to be trained; and determine the image feature vector of each image sample based on the random image sample set and the image data of each image sample in at least one target defect image sample.

[0075] The generator is used to generate random images. The generator to be trained refers to a generator that has not yet been trained and cannot generate random images with the defect type matching the target defect type. In the initial stage of training, the generator to be trained generates a certain number of random images based on its initial parameters. In one embodiment, the generator's network model consists of a fully connected neural network and a deconvolutional network. The random image sample set generated through Gaussian distributed random noise specifically includes:

[0076] Random sampling based on Gaussian distribution yields multiple noise vectors;

[0077] Based on multiple noise vectors, a pixel matrix of a preset size is generated, and each pixel matrix is ​​used as a random image sample.

[0078] The preset size is the set resolution of the defective image; for example, the preset size can be 512*512 or 256*256. Multiple noise vectors are obtained by randomly sampling from a Gaussian distribution using the θ parameter. These noise vectors are then input into the generator's network model to obtain random image samples of the preset size.

[0079] Specifically, in the design of the generator's fully connected network model and deconvolutional network model, the number of hidden layers and convolutional kernels is related to the preset size. For example... Figure 3 The generator fully connected network model structure shown is as follows: Figure 4 The generator convolutional network model structure shown is designed based on a preset size of 512*512. Figure 3 In the middle, X n The noise vector is randomly sampled from a Gaussian distribution controlled by the parameter θ. The input layer has 64 nodes, with the ReLU activation function for forward propagation; the first layer has 128 nodes, with the ReLU activation function for forward propagation; the second layer has 256 nodes, with the ReLU activation function for forward propagation; the third layer has 512 nodes, with the Sigmoid activation function for forward propagation; the output layer has 1024 nodes. Figure 3 In this model, g = 1024. The output layer result ranges from [0, 1], and is flattened using reshape to form a 32*32 matrix. After obtaining the output result of the fully connected network model, a microstepping convolutional network is used to obtain the final pixel matrix. For example... Figure 4The convolutional network model is designed as follows: Layer 1: 3 x 4 convolutional kernels, stride s = 1 / 2, padding = 2; Layer 2: 9 x 4 convolutional kernels, stride s = 1 / 2, padding = 2; Layer 3: 9 x 4 convolutional kernels, stride s = 1 / 2, padding = 2; Layer 4: 3 x 2 convolutional kernels, stride s = 1 / 2, padding = 1. This yields a 512x512 pixel matrix, which can be used as a random image sample.

[0080] Image data refers to data describing the physical properties of an image itself, such as the pixel values. An image feature vector is a vector composed of parameters describing image features. It includes multiple image features, each determined by feature extraction from the image sample's data. Image features are the most fundamental characteristics that distinguish one image from another; they are its defining attributes. Therefore, an image feature vector can characterize the attributes of an image sample.

[0081] Step 206: Input the image feature vector of each image sample in the random image sample set and at least one target defect image sample into the first discriminator to obtain the first loss function value of the first discriminator. Based on the first loss function value, perform the first training on the generator to be trained and the first discriminator to obtain the random image sample set that meets the preset conditions, the generator to be trained after the first training, and the first discriminator.

[0082] The first discriminator compares the random image sample set with the target defect image sample, and uses a predefined method to make the random image sample generated by the generator approximate the features of the target defect image sample as closely as possible. Specifically, after obtaining the image feature vector of each image sample in the random image sample set and at least one target defect image sample, the first discriminator can use a multilayer perceptron to further analyze the features to obtain the correlation between the image features and the true or false image result.

[0083] In one embodiment, the network model of the first discriminator is as follows: Figure 5 As shown, the image feature vector A of each image sample is... n After being input into the network model of the first discriminator, the output value B is obtained after passing through the hidden layer structure. In one embodiment, Figure 5The network structure shown includes four hidden layers: the first layer has 96 nodes with the ReLU activation function for forward propagation; the second layer has 48 nodes with the ReLU activation function for forward propagation; the third layer has 24 nodes with the ReLU activation function for forward propagation; and the fourth layer has 12 nodes with the ReLU activation function for forward propagation. The target defect image sample is used as the real image, and the output value B represents the difference between the image features of the random image sample and the features of the target defect image sample. After obtaining the output value B, the first loss function value of the first discriminator is calculated based on the output value B. The generator and the first discriminator are trained using the first loss function value. During training, the network model parameters of the generator and the first discriminator are changed to ensure that the generated random image samples learn the image features of the target defect image sample as much as possible.

[0084] The preset conditions are used to end the training loop. These conditions can be based on the quality of the training results, the number of training iterations, or the training duration. By defining these preset conditions, the desired random image samples are obtained. These random image samples, to a certain extent, learn the image features of the target defect image samples.

[0085] Step 208: Obtain the semantic description feature vector of each image sample in the random image sample set and the defect image sample set that meet the preset conditions. Input the semantic description feature vector into the second discriminator to obtain the second loss function value of the second discriminator. Based on the first loss function value and the second loss function value, perform a second training on the generator to be trained, the first discriminator, and the second discriminator after the first training to obtain the generator to be trained, the first discriminator, and the second discriminator after the second training.

[0086] The semantic description feature vector includes multiple semantic description features, which characterize the product's appearance defects, such as the color, outline, and shape of the defects. Each semantic description feature has corresponding discrimination conditions. In some embodiments, to reduce the difficulty of feature discrimination and improve the subsequent model training effect, the discrimination conditions for each semantic description feature are set to only two states: met or not met. Setting classification criteria requires ensuring that the semantic description features of different defects have at least one distinguishing feature. The table below shows an example of the semantic description feature extraction results for a batch of PCB products. According to the definition rules of semantic description feature vectors, a total of 3 discrimination rules are defined in this scenario. Therefore, each defect type corresponds to one 3-dimensional semantic description feature vector. For example, the feature vector of defect 1 is (1, 0, 1). T .

[0087] Table 1 Examples of semantic feature evaluation of defect images

[0088] Is it a continuous contour? Are the holes missing? Are the lines and outlines smooth? Defect 1 √ × √ Defect 2 × √ √ Defect 3 √ × ×

[0089] The semantic description feature vector of each image sample is obtained through a feature extraction model. This application does not specifically limit the structure of the feature extraction model of the semantic description feature vector. It can be the same as or different from the feature extraction model of the image feature vector.

[0090] The second discriminator primarily helps the generator learn various semantic description features as accurately as possible, ensuring that the generated images meet the semantic description requirements and enhancing the quality of the generated images. The network model structure of the second discriminator is a classifier model, which can be the same as or different from the network model structure of the first discriminator. After inputting the semantic description feature vectors of each image sample from a set of random image samples and a set of defective image samples that meet preset conditions into the second discriminator, the classification results of the semantic description features of the random image samples are obtained. Based on the classification results, the second loss function value of the second discriminator is calculated. The model parameters of the generator, the first discriminator, and the second discriminator are then adjusted based on the second loss function value, and training is performed until the loss function value meets the conditions. Specifically, the first and second loss functions can be combined for a second training iteration.

[0091] Step 210: Based on the generator to be trained after the second training, the first discriminator, and the second discriminator, generate random image samples with the defect type of the target defect type.

[0092] After optimizing the generator, the first discriminator, and the second discriminator based on the target defect type, noise can be randomly generated multiple times in a Gaussian distribution, and PCB image samples of the target defect type can be generated using the optimized generator.

[0093] In the method provided in the above embodiments, a defect image sample set is obtained; based on the defect type of each image sample in the defect image sample set, the target defect type and at least one corresponding target defect image sample are determined; a random image sample set generated by the generator to be trained is obtained; based on the random image sample set and the image data of each image sample in the at least one target defect image sample, the image feature vector of each image sample is determined; the random image sample set and the image feature vector of each image sample in the at least one target defect image sample are input into a first discriminator to obtain a first loss function value of the first discriminator; based on the first loss function value, the generator to be trained and the first discriminator are trained for the first time to obtain a result that meets the preset criteria. The system uses a random image sample set, a generator to be trained after the first training, and a first discriminator. It acquires the semantic description feature vector of each image sample from the random image sample set and the defect image sample set that meet preset conditions. This semantic description feature vector is input into a second discriminator to obtain its second loss function value. Based on the first and second loss function values, the generator to be trained after the first training, the first discriminator, and the second discriminator are trained a second time to obtain the generator to be trained after the second training, the first discriminator, and the second discriminator. Based on the generator to be trained after the second training, the first discriminator, and the second discriminator, random image samples with the target defect type are generated. The extraction and fusion of semantic description features based on defect type improves the accuracy of defect feature recognition in defect images. By fusing semantic description features to train the generator and discriminator models, the defect image samples of small sample products are expanded, improving the efficiency and accuracy of real-time defect recognition.

[0094] In one embodiment, determining the target defect type and at least one corresponding target defect image sample based on the defect type of each image sample in the defect image sample set includes:

[0095] Based on the defect type of each image sample in the defect image sample set, count the number of image samples of each defect type in the defect image sample set.

[0096] The defect type with the smallest number of image samples and not less than 1 is selected as the target defect type.

[0097] Based on the learning process of defect detection models, it is known that the fewer the number of training samples, the more prone the model is to coupling, and the worse the training effect will be. Therefore, in a known set of defect image samples, the target defect type with the fewest image samples is determined by identifying the defect type of each image sample. It can be understood that the target defect type is the defect type that most needs to generate image samples through the generator. Furthermore, when determining the target defect type based on the number of image samples, it is necessary to ensure that there are multiple image samples of the target defect type.

[0098] For example, in one embodiment, the defect image sample set contains 1,500 image samples, which have a total of 9 defect types. According to statistics, the number of image samples for the 9 defect types are 59, 67, 124, 237, 46, 15, 489, 330, and 133, respectively. Then, the 6th defect type with 15 image samples is taken as the target defect type.

[0099] In the method provided in the above embodiments, by determining the defect type with the fewest image samples, and using this defect type as the target defect type for optimization training of the generator, image samples of the target defect type can be obtained, thereby improving the detection effect of the defect detection model on the target defect type.

[0100] In one embodiment, such as Figure 6 As shown, the image data includes image size and pixel values; correspondingly, based on the image data of each image sample in the random image sample set and at least one target defect image sample, the image feature vector of each image sample is determined, including:

[0101] Step 602: For any image sample in the random image sample set and at least one target defect image sample, determine whether the image size of any image sample is a preset size. If the image size of any image sample is not a preset size, perform size compensation or image cropping on any image sample to obtain any preprocessed image sample.

[0102] Step 604: Normalize the pixel values ​​of any preprocessed image sample, extract the features of any image sample after pixel normalization, and obtain the image feature vector of any image sample.

[0103] The preset size is a predefined size of image samples that the generator and discriminator models can process. The size of the preset size affects the design of the generator and discriminator model structures. Therefore, if the generator and discriminator network models have already been built, the preset size is known and fixed.

[0104] Before extracting the image feature vectors of the image samples, preprocessing is required to ensure the proper execution of the generator and discriminator and the consistency of the results. Specifically, the image samples undergo size compensation and format normalization, and grayscale pixels are added to form an RGB image of a preset size without distortion. Then, the defective image samples are normalized so that all pixel values ​​of the defective image samples are normalized to the range of 0 to 1.

[0105] Image feature extraction network models can be constructed by building multi-scale fusion feature extraction networks to extract image features. The network model construction process mainly involves setting the number of network layers, the number and structure of residual blocks, and global average pooling to build the image feature extraction model. In one embodiment, the image feature extraction network model design structure is as follows:

[0106] The first layer is a 3*3 convolutional layer with 64 convolutional kernels, a stride of 2, padding of 1, and the activation function is ReLU.

[0107] The second layer is a max pooling layer with a size of 3*3 and a step size of 2.

[0108] The third layer consists of 4 residual blocks, each containing 3 residual blocks, for a total of 12 residual blocks;

[0109] The fourth layer consists of 4 residual blocks, each containing 20 residual blocks, for a total of 80 residual blocks;

[0110] The last layer is a global average pooling layer, which performs average pooling on the output features to obtain the image feature vector.

[0111] In the method provided in the above embodiments, by performing preprocessing operations on image samples and standardizing the feature extraction of all image samples, effective image feature vectors can be obtained, thereby ensuring the optimization effect of the generator.

[0112] In one embodiment, the image feature vector of each image sample from a random image sample set and at least one target defect image sample is input into a first discriminator to obtain a first loss function of the first discriminator, including:

[0113] The image feature vectors of all image samples in the random image sample set and at least one target defect image sample are input into the first discriminator to obtain the true or false result of each image sample in the random image sample set;

[0114] The first loss function value is calculated based on the true / false results of all image samples in the random image sample set and the manual annotation results of at least one target defect image sample.

[0115] It should be noted that the result of at least one target defect image sample is preset through manual annotation. In one embodiment, if the output of the first discriminator is 1*3, the manual annotation result of the target defect image sample can be set to (0,0,0). The true and false results are the discrimination output results of the first discriminator for each random image sample. The first loss function value is determined by comparing the true and false results of all image samples in the random image sample set with the manual annotation result of at least one target defect image sample.

[0116] Specifically, the first loss function value can be obtained by calculating the respective distributions of a random image sample set and a set of at least one target defect image sample set, and then calculating the Wassertain distance between these two distributions. The Wassertain distance is calculated as follows:

[0117]

[0118] Among them (P) r ,P g ) is P r and P g The set of all possible joint distributions. For each possible joint distribution γ, we can sample (x,y) ~ γ to obtain a true sample x and a generated sample y, and calculate the distance ‖xy‖ between these two samples. Therefore, we can calculate the expected value E for a specific sample under this joint distribution γ. (x,y)～γ [‖xy‖]. The lower bound reached by taking this expectation value among all possible joint distributions. This is the Wassertein distance.

[0119] Since the lower bound is difficult to solve in practice, in some embodiments the loss function is transformed into the following form:

[0120]

[0121] Where sup is the upper bound of the expression, K is a constant, and the following expression must hold for any two numbers x1 and x2 in the domain:

[0122] |f(x1)-f(x2)|≤K|x1-x2|

[0123] In the method provided in the above embodiments, a first loss function is calculated based on true and false results, so that the generator can generate an image that approximates the features of the defective image sample as closely as possible through the set first loss function.

[0124] In one embodiment, the generator and the first discriminator are trained for the first time based on the first loss function value, resulting in a random image sample set that meets preset conditions, the generator to be trained after the first training, and the first discriminator, including:

[0125] If the first loss function value is not less than the preset threshold, adjust the optimization parameters of the generator to be trained and the first discriminator, return to the step of obtaining the random image sample set generated by the generator to be trained, continue to train the generator to be trained and the first discriminator for the first time, and count the number of loops.

[0126] If the first loss function value is less than a preset threshold or the number of iterations is greater than a preset number, a random image sample set that meets the preset conditions, a generator to be trained after the first training, and a first discriminator are obtained.

[0127] The magnitude of the first loss function value characterizes the feature difference between the random image sample and the target defect image sample. Therefore, if the first loss function value is less than the preset threshold or the number of iterations is greater than the preset number of iterations, the random image sample is imported into the second discriminator for discrimination. Otherwise, the optimization parameters of the generator to be trained and the first discriminator are adjusted, and the loop continues.

[0128] In the method provided in the above embodiments, by setting a first loss function and conditions corresponding to the number of iterations, the first training of the generator to be trained and the first discriminator is terminated, which can obtain a random image sample set that meets the preset conditions in an effective time, thereby improving training efficiency.

[0129] In one embodiment, such as Figure 7 As shown, the second discriminator includes multiple semantic feature sub-discriminators. These multiple semantic feature sub-discriminators are used to determine the semantic feature type of the image samples. The semantic description feature vector is input into the second discriminator to obtain the second loss function value of the second discriminator, including:

[0130] Step 702: Input the semantic description feature vector into each semantic feature sub-discriminator to obtain each discrimination result;

[0131] Step 704: Calculate the second loss function value of the second discriminator based on the number of target discrimination results in each discrimination result.

[0132] The second discriminator sets up sub-discriminators based on each semantic description feature, and finally merges all sub-discriminators to form the final second discriminator. The semantic description feature vector is input into each semantic feature sub-discriminator to obtain the discrimination result corresponding to each semantic description feature. The output of each sub-discriminator represents the discrimination result of a single semantic description feature, taking a value of 0 or 1. Then, the correctness of each discrimination result is determined by manual annotation. By judging the number of correct discrimination results, the second loss function value of the second discriminator is calculated, as follows:

[0133] P2 = 1 - c / n

[0134] Where c is the number of correctly judged results, and n is the total number of semantic description features.

[0135] In the method provided in the above embodiments, the discrimination effect of the second discriminator can be quickly and accurately determined by the number of discrimination results, thereby determining the quality of the random image generated by the generator.

[0136] In one embodiment, based on the first loss function value and the second loss function value, the generator to be trained after the first training, the first discriminator, and the second discriminator are trained a second time to obtain the generator to be trained after the second training, the first discriminator, and the second discriminator, including:

[0137] The fusion loss function value is determined based on the first loss function value and the second loss function value.

[0138] Based on the fusion loss function value, the generator to be trained, the first discriminator, and the second discriminator after the first training are trained a second time to obtain the generator to be trained, the first discriminator, and the second discriminator after the second training.

[0139] The fusion loss function value is obtained by fusing the first and second loss function values. Before fusion, the first and second loss function values ​​need to be standardized. Specifically, the first loss function is standardized to a predicted score value P1, with a value in the range [0,1]. The closer the value is to 0, the greater the difference between the random image sample and the defective image sample; the closer it is to 1, the closer the difference between the random image sample and the defective image sample. The fusion loss function value for the second training process is:

[0140] P = α1P2 + α2P2

[0141] Where α1 and α2 are fractional weights, with values ​​ranging from [0,1].

[0142] During the fusion training of the first discriminator and the second discriminator, P is used as the score of the fusion loss function. The optimization strategy can be to update the P value by calculating the model gradient so that the P value increases at the fastest rate, thereby completing the second training of the generator to be trained, the first discriminator, and the second discriminator.

[0143] In the method provided in the above embodiments, the second training is completed by fusing the loss function values, which can effectively combine the training of the first discriminator and the second discriminator, comprehensively consider the discrimination effects of the two discriminators, and improve the optimization training effect of the generator to be trained.

[0144] In one embodiment, such as Figure 8 As shown, a defect image generation method incorporating semantic rules is provided, including:

[0145] Step 1: Randomly extract noise vectors using a Gaussian distribution;

[0146] Step 2: Input the noise vector into the generator to generate random image samples;

[0147] Step 3: Obtain target defect image samples with the defect type of the target defect type, and extract image features from the random image samples and the target defect image samples to obtain image feature vectors;

[0148] Step 4: Input the image feature vector into discriminator 1 to obtain the first loss function value of discriminator 1;

[0149] Step 5: When the first loss function value is less than the preset threshold θ1 or the number of iterations is greater than the preset number of iterations N, input the random image sample into the discriminator 2 and execute step 6; otherwise, optimize the generator and discriminator 1 and jump to step 2.

[0150] Step 6: Obtain the semantic description feature vector of each image sample in the defective image sample set, and input the semantic description feature vector into discriminator 2 to obtain the second loss function value of discriminator 2;

[0151] Step 7: Determine the fusion loss function value based on the second loss function value. If the fusion loss function value does not meet the conditions, optimize the generator, discriminator 1 and discriminator 2, and jump to step 2. Continue until the fusion loss function value meets the conditions to obtain the optimized generator, which is used to generate image samples with the defect type of the target defect type.

[0152] The method provided in the above embodiments can expand the scarce defect type samples of PCB products in the early stage of production, providing sufficient training samples for their visual recognition model; the discrimination method based on loss function and semantic description features can help the model more accurately distinguish whether the product defect belongs to that type of defect in the subsequent recognition process.

[0153] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0154] Based on the same inventive concept, this application also provides an image sample generation apparatus for implementing the image sample generation method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more image sample generation apparatus embodiments provided below can be found in the limitations of the image sample generation method described above, and will not be repeated here.

[0155] In one embodiment, such as Figure 9 As shown, an image sample generation device is provided, comprising: a sample acquisition module 901, a feature extraction module 902, a first discrimination module 903, a second discrimination module 904, and a sample generation module 905, wherein:

[0156] The sample acquisition module 901 is used to acquire a defect image sample set, and determine the target defect type and at least one corresponding target defect image sample based on the defect type of each image sample in the defect image sample set.

[0157] The feature extraction module 902 is used to obtain a random image sample set generated by the generator to be trained, and to determine the image feature vector of each image sample based on the random image sample set and the image data of each image sample in at least one target defect image sample.

[0158] The first discrimination module 903 is used to input the image feature vector of each image sample in the random image sample set and at least one target defect image sample into the first discriminator to obtain the first loss function value of the first discriminator, and perform the first training on the generator to be trained and the first discriminator based on the first loss function value to obtain the random image sample set that meets the preset conditions, the generator to be trained after the first training and the first discriminator.

[0159] The second discrimination module 904 is used to obtain the semantic description feature vector of each image sample in the random image sample set and the defect image sample set that meet the preset conditions, input the semantic description feature vector into the second discriminator, obtain the second loss function value of the second discriminator, and perform a second training on the generator to be trained, the first discriminator and the second discriminator after the first training based on the first loss function value and the second loss function value, to obtain the generator to be trained, the first discriminator and the second discriminator after the second training.

[0160] The sample generation module 905 is used to generate random image samples with the defect type of the target defect type based on the generator to be trained after the second training, the first discriminator and the second discriminator.

[0161] In one embodiment, the sample acquisition module 901 is further configured to:

[0162] Based on the defect type of each image sample in the defect image sample set, count the number of image samples of each defect type in the defect image sample set.

[0163] The defect type with the smallest number of image samples and not less than 1 is selected as the target defect type.

[0164] In one embodiment, the feature extraction module 902 is further configured to:

[0165] Random sampling based on Gaussian distribution yields multiple noise vectors;

[0166] Based on multiple noise vectors, a pixel matrix of a preset size is generated, and each pixel matrix is ​​used as a random image sample.

[0167] In one embodiment, the feature extraction module 902 is further configured to:

[0168] For any image sample in the random image sample set and at least one target defect image sample, determine whether the image size of any image sample is a preset size. If the image size of any image sample is not a preset size, perform size compensation or image cropping on any image sample to obtain any preprocessed image sample.

[0169] The pixel values ​​of any image sample after preprocessing are normalized, and the features of any image sample after pixel normalization are extracted to obtain the image feature vector of any image sample.

[0170] In one embodiment, the first discrimination module 903 is further configured to:

[0171] The image feature vectors of all image samples in the random image sample set and at least one target defect image sample are input into the first discriminator to obtain the true or false result of each image sample in the random image sample set;

[0172] The first loss function value is calculated based on the true / false results of all image samples in the random image sample set and the manual annotation results of at least one target defect image sample.

[0173] In one embodiment, the first discrimination module 903 is further configured to:

[0174] If the first loss function value is not less than the preset threshold, adjust the optimization parameters of the generator to be trained and the first discriminator, return to the step of obtaining the random image sample set generated by the generator to be trained, continue to train the generator to be trained and the first discriminator for the first time, and count the number of loops.

[0175] If the first loss function value is less than a preset threshold or the number of iterations is greater than a preset number, a random image sample set that meets the preset conditions, a generator to be trained after the first training, and a first discriminator are obtained.

[0176] In one embodiment, the second discrimination module 904 is further configured to:

[0177] The semantic description feature vectors are input into each semantic feature sub-discriminator to obtain each discrimination result;

[0178] The second loss function value of the second discriminator is calculated based on the number of target discrimination results in each discrimination result.

[0179] In one embodiment, the second discrimination module 904 is further configured to:

[0180] The fusion loss function value is determined based on the first loss function value and the second loss function value.

[0181] Based on the fusion loss function value, the generator to be trained, the first discriminator, and the second discriminator after the first training are trained a second time to obtain the generator to be trained, the first discriminator, and the second discriminator after the second training.

[0182] Each module in the aforementioned image sample generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0183] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores model data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements an image sample generation method.

[0184] Those skilled in the art will understand that Figure 10The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0185] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0186] Obtain a defect image sample set, and determine the target defect type and at least one corresponding target defect image sample based on the defect type of each image sample in the defect image sample set;

[0187] Obtain a set of random image samples generated by the generator to be trained, and determine the image feature vector of each image sample based on the random image sample set and the image data of each image sample in at least one target defect image sample.

[0188] The image feature vector of each image sample in the random image sample set and at least one target defect image sample is input into the first discriminator to obtain the first loss function value of the first discriminator. Based on the first loss function value, the generator to be trained and the first discriminator are trained for the first time to obtain the random image sample set that meets the preset conditions, the generator to be trained after the first training and the first discriminator.

[0189] Obtain the semantic description feature vector of each image sample in the random image sample set and the defect image sample set that meet the preset conditions. Input the semantic description feature vector into the second discriminator to obtain the second loss function value of the second discriminator. Based on the first loss function value and the second loss function value, perform a second training on the generator to be trained, the first discriminator, and the second discriminator after the first training to obtain the generator to be trained, the first discriminator, and the second discriminator after the second training.

[0190] Based on the generator to be trained after the second training, the first discriminator, and the second discriminator, random image samples with the defect type of the target defect type are generated.

[0191] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0192] Based on the defect type of each image sample in the defect image sample set, count the number of image samples of each defect type in the defect image sample set.

[0193] The defect type with the smallest number of image samples and not less than 1 is selected as the target defect type.

[0194] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0195] Random sampling based on Gaussian distribution yields multiple noise vectors;

[0196] Based on multiple noise vectors, a pixel matrix of a preset size is generated, and each pixel matrix is ​​used as a random image sample.

[0197] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0198] For any image sample in the random image sample set and at least one target defect image sample, determine whether the image size of any image sample is a preset size. If the image size of any image sample is not a preset size, perform size compensation or image cropping on any image sample to obtain any preprocessed image sample.

[0199] The pixel values ​​of any image sample after preprocessing are normalized, and the features of any image sample after pixel normalization are extracted to obtain the image feature vector of any image sample.

[0200] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0201] The image feature vectors of all image samples in the random image sample set and at least one target defect image sample are input into the first discriminator to obtain the true or false result of each image sample in the random image sample set;

[0202] The first loss function value is calculated based on the true / false results of all image samples in the random image sample set and the manual annotation results of at least one target defect image sample.

[0203] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0204] If the first loss function value is not less than the preset threshold, adjust the optimization parameters of the generator to be trained and the first discriminator, return to the step of obtaining the random image sample set generated by the generator to be trained, continue to train the generator to be trained and the first discriminator for the first time, and count the number of loops.

[0205] If the first loss function is less than a preset threshold or the number of iterations is greater than a preset number, a random image sample set that meets the preset conditions, a generator to be trained after the first training, and a first discriminator are obtained.

[0206] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0207] The semantic description feature vectors are input into each semantic feature sub-discriminator to obtain each discrimination result;

[0208] The second loss function value of the second discriminator is calculated based on the number of target discrimination results in each discrimination result.

[0209] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0210] The fusion loss function value is determined based on the first loss function value and the second loss function value.

[0211] Based on the fusion loss function value, the generator to be trained, the first discriminator, and the second discriminator after the first training are trained a second time to obtain the generator to be trained, the first discriminator, and the second discriminator after the second training.

[0212] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0213] Obtain a defect image sample set, and determine the target defect type and at least one corresponding target defect image sample based on the defect type of each image sample in the defect image sample set;

[0214] Obtain a set of random image samples generated by the generator to be trained, and determine the image feature vector of each image sample based on the random image sample set and the image data of each image sample in at least one target defect image sample.

[0215] The image feature vector of each image sample in the random image sample set and at least one target defect image sample is input into the first discriminator to obtain the first loss function value of the first discriminator. Based on the first loss function value, the generator to be trained and the first discriminator are trained for the first time to obtain the random image sample set that meets the preset conditions, the generator to be trained after the first training and the first discriminator.

[0216] Obtain the semantic description feature vector of each image sample in the random image sample set and the defect image sample set that meet the preset conditions. Input the semantic description feature vector into the second discriminator to obtain the second loss function value of the second discriminator. Based on the first loss function value and the second loss function value, perform a second training on the generator to be trained, the first discriminator, and the second discriminator after the first training to obtain the generator to be trained, the first discriminator, and the second discriminator after the second training.

[0217] Based on the generator to be trained after the second training, the first discriminator, and the second discriminator, random image samples with the defect type of the target defect type are generated.

[0218] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0219] Based on the defect type of each image sample in the defect image sample set, count the number of image samples of each defect type in the defect image sample set.

[0220] The defect type with the smallest number of image samples and not less than 1 is selected as the target defect type.

[0221] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0222] Random sampling based on Gaussian distribution yields multiple noise vectors;

[0223] Based on multiple noise vectors, a pixel matrix of a preset size is generated, and each pixel matrix is ​​used as a random image sample.

[0224] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0225] For any image sample in the random image sample set and at least one target defect image sample, determine whether the image size of any image sample is a preset size. If the image size of any image sample is not a preset size, perform size compensation or image cropping on any image sample to obtain any preprocessed image sample.

[0226] The pixel values ​​of any image sample after preprocessing are normalized, and the features of any image sample after pixel normalization are extracted to obtain the image feature vector of any image sample.

[0227] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0228] The image feature vectors of all image samples in the random image sample set and at least one target defect image sample are input into the first discriminator to obtain the true or false result of each image sample in the random image sample set;

[0229] The first loss function value is calculated based on the true / false results of all image samples in the random image sample set and the manual annotation results of at least one target defect image sample.

[0230] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0231] If the first loss function value is not less than the preset threshold, adjust the optimization parameters of the generator to be trained and the first discriminator, return to the step of obtaining the random image sample set generated by the generator to be trained, continue to train the generator to be trained and the first discriminator for the first time, and count the number of loops.

[0232] If the first loss function is less than a preset threshold or the number of iterations is greater than a preset number, a random image sample set that meets the preset conditions, a generator to be trained after the first training, and a first discriminator are obtained.

[0233] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0234] The semantic description feature vectors are input into each semantic feature sub-discriminator to obtain each discrimination result;

[0235] The second loss function value of the second discriminator is calculated based on the number of target discrimination results in each discrimination result.

[0236] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0237] The fusion loss function value is determined based on the first loss function value and the second loss function value.

[0238] Based on the fusion loss function value, the generator to be trained, the first discriminator, and the second discriminator after the first training are trained a second time to obtain the generator to be trained, the first discriminator, and the second discriminator after the second training.

[0239] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements all the steps in the above embodiments.

[0240] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0241] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0242] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image sample generation method, characterized by, The method comprises: acquiring a defect image sample set, determining a target defect type and at least one target defect image sample corresponding to the target defect type according to a defect type of each image sample in the defect image sample set; acquiring a random image sample set generated by a to-be-trained generator, determining an image feature vector of each image sample in the random image sample set and the at least one target defect image sample according to image data of each image sample; inputting the image feature vector of each image sample in the random image sample set and the at least one target defect image sample into a first discriminator to obtain a first loss function value of the first discriminator, performing first training on the to-be-trained generator and the first discriminator based on the first loss function value, and obtaining a random image sample set meeting a preset condition, the to-be-trained generator after the first training, and the first discriminator; acquiring a semantic description feature vector of each image sample in the defect image sample set and the random image sample set meeting the preset condition, inputting the semantic description feature vector into a second discriminator to obtain a second loss function value of the second discriminator, determining a fusion loss function value based on the first loss function value and the second loss function value, and performing second training on the to-be-trained generator, the first discriminator, and the second discriminator after the first training based on the fusion loss function value to obtain the to-be-trained generator, the first discriminator, and the second discriminator after the second training; generating a random image sample with a defect type of the target defect type based on the to-be-trained generator, the first discriminator, and the second discriminator after the second training.

2. The method of claim 1, wherein, The method comprises: counting the number of image samples of each defect type in the defect image sample set according to the defect type of each image sample in the defect image sample set; taking a defect type with the smallest number of image samples and not less than 1 as the target defect type.

3. The method of claim 1, wherein, The method comprises: performing random sampling based on a Gaussian distribution to obtain a plurality of noise vectors; generating a pixel matrix of a preset size based on the plurality of noise vectors, and taking each pixel matrix as a random image sample.

4. The method of claim 1, wherein, The image data comprises an image size and a pixel value; accordingly, the method comprises: judging whether the image size of any image sample in the random image sample set and the at least one target defect image sample is the preset size, performing size compensation or image cropping on the any image sample in a case where the image size of the any image sample is not the preset size to obtain a preprocessed any image sample; The pixel value of any pre-processed image sample is normalized, and the feature of the pixel normalized any image sample is extracted to obtain an image feature vector of the any image sample.

5. The method of claim 1, wherein, The image feature vector of each image sample in the random image sample set and the at least one target defect image sample is input into the first discriminator to obtain a first loss function of the first discriminator, including: The image feature vector of each image sample in the random image sample set and the at least one target defect image sample is input into the first discriminator to obtain a true or false result of each image sample in the random image sample set; The first loss function value is calculated according to the true or false result of all image samples in the random image sample set and the artificial labeling result of the at least one target defect image sample.

6. The method of claim 5, wherein, The first loss function value is calculated according to the true or false result of all image samples in the random image sample set and the artificial labeling result of the at least one target defect image sample. The first loss function value is calculated according to the true or false result of all image samples in the random image sample set and the artificial labeling result of the at least one target defect image sample. In the case that the first loss function value is not less than the preset threshold, the optimization parameters of the training generator and the first discriminator are adjusted, and the step of obtaining the random image sample set generated by the training generator is returned to continue the first training of the training generator and the first discriminator, and the number of cycles is counted; 7. The method of claim 1, wherein, In the case that the first loss function value is less than the preset threshold or the number of cycles is greater than the preset number, the random image sample set meeting the preset condition, the training generator after the first training and the first discriminator are obtained. The second discriminator includes a plurality of semantic feature sub discriminators, and the plurality of semantic feature sub discriminators are used to determine the semantic feature type of the image sample. The semantic description feature vector is input into each semantic feature sub discriminator to obtain each discrimination result.

8. An image sample generation apparatus characterized by comprising: The second loss function value of the second discriminator is calculated according to the number of target discrimination results in each discrimination result. The device includes: The sample acquisition module is configured to acquire a defect image sample set, determine a target defect type and at least one target defect image sample corresponding to the target defect type according to the defect type of each image sample in the defect image sample set; The feature extraction module is configured to acquire a random image sample set generated by a training generator, and determine an image feature vector of each image sample according to image data of each image sample in the random image sample set and the at least one target defect image sample; The first discrimination module is configured to input image feature vectors of each image sample in the random image sample set and the at least one target defect image sample into a first discriminator to obtain a first loss function value of the first discriminator, perform first training on the to-be-trained generator and the first discriminator based on the first loss function value, and obtain a random image sample set meeting a preset condition, the to-be-trained generator after the first training, and the first discriminator; The second discrimination module is configured to obtain semantic description feature vectors of each image sample in the random image sample set meeting the preset condition and the defect image sample set, input the semantic description feature vectors into a second discriminator to obtain a second loss function value of the second discriminator, determine a fusion loss function value based on the first loss function value and the second loss function value, perform second training on the to-be-trained generator, the first discriminator after the first training, and the second discriminator according to the fusion loss function value, and obtain the to-be-trained generator, the first discriminator, and the second discriminator after the second training. The sample generation module is configured to generate a random image sample of a target defect type according to the to-be-trained generator, the first discriminator, and the second discriminator after the second training. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 7.

11. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 7.

Images