Image generation system and image generation method
Through the image generation system, the conditional generation adversarial network is used to generate high-quality defect images, which solves the problem of expanding defect image data sets in industrial visual defect detection, and improves model performance and training stability.
Patent Information
- Application Number
- CN202411487811.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-23
- Publication Date
- 2025-02-14
AI Technical Summary
The prior art has problems such as high acquisition costs, insufficient data quantity, and unbalanced defect categories in the expansion of defect image data sets in industrial vision defect detection, which limits the performance of artificial intelligence models.
An image generation system is proposed to generate a third image for providing conditional information associated with image defects by acquiring a first image set including defective images and normal maps and a second image set without defects. Then, based on the condition, an adversarial network is generated, and a fourth image is generated using the third image, which contains image defects for augmenting the defective image data set.
By generating more diverse and high-quality defect images, the diversity and authenticity of the data set are improved, and the performance and training stability of the artificial intelligence model are enhanced.
Smart Images
Figure CN119444931A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing, and in particular to an image generation system and an image generation method. Background Art
[0002] In the scenario of industrial visual defect detection, deep learning-based artificial intelligence models (such as defect detection models) are increasingly being used. For deep learning-based artificial intelligence models, it is necessary to train them with defect image datasets. However, defect image datasets have problems such as high acquisition cost, insufficient overall quantity, missing defect categories or unbalanced quantity, which will limit the performance of artificial intelligence models.
[0003] However, there is currently no solution for how to expand the relevant defect image dataset to provide the model with a higher quality defect image training set. Summary of the invention
[0004] In order to solve the related technical problems, the embodiments of the present application provide an image generation system and an image generation method.
[0005] The technical solution of the embodiment of the present application is implemented as follows:
[0006] The embodiment of the present application provides an image generation system, including: a first module and a second module; wherein,
[0007] The first module is used to obtain a first image set and a second image set, wherein the first image set includes one or more first images and corresponding normal maps, each of which includes an image defect, and the second image set includes one or more second images, each of which does not include an image defect; and generate one or more third images using the first image set and the second image set, wherein the third images are used to provide condition information associated with the image defect;
[0008] The second module is used to generate one or more fourth images based on a conditional generative adversarial network using the one or more third images, each of the fourth images containing one or more image defects.
[0009] In the above system, the second module is used to obtain one or more noise vectors; based on a conditional generative adversarial network, the one or more third images and the one or more noise vectors are used to generate the one or more fourth images.
[0010] In the above system, the second module is also used to obtain one or more real images; based on the one or more real images and the one or more fourth images, adjust the network parameters of the conditional generative adversarial network so that the conditional generative adversarial network has a convergence direction.
[0011] In the above system, the second module is used to classify the one or more fourth images using the one or more real images to obtain a classification result, and the classification result characterizes that the one or more fourth images are real images or forged images; and use the classification result to adjust the network parameters of the conditional generative adversarial network.
[0012] In the above system, the second module is used to classify the one or more fourth images using the one or more real images and the one or more third images to obtain classification results; use the classification results to calculate the loss value of the conditional generative adversarial network, and use the loss value to adjust the network parameters of the conditional generative adversarial network.
[0013] In the above system, the first module is used to select K first images and normal maps corresponding to the K first images from the first image set, and select a second image from the second image set, where K is an integer greater than or equal to 1; and fuse the K first images, the normal maps corresponding to the K first images, and the selected second image to obtain a third image.
[0014] In the above system, the first module is used to use the normal maps corresponding to the K first images to calculate the light intensity characteristics of the K first images; and to fuse the K first images, the light intensity characteristics of the K first images and the selected second image to obtain a third image.
[0015] In the above system, the first module is used to convert each pixel in the normal map corresponding to each first image of the K first images in turn to obtain the normal vector corresponding to the first image; and use the normal vector corresponding to the first image to calculate the light intensity characteristics of the first image.
[0016] In the above system, the system further includes: a third module; wherein,
[0017] The third module is used to use one or more real images and the one or more fourth images to train the first model and the second model respectively, to obtain the trained first model and the second model; use the trained first model and the second model to perform defect detection on one or more test images, to obtain the first detection result and the second detection result respectively; use the first detection result and the second detection result to determine whether the one or more fourth images meet the requirements.
[0018] The embodiment of the present application also provides an image generation method, which is applied to an image generation system, including:
[0019] The first module of the image generation system acquires a first image set and a second image set, wherein the first image set includes one or more first images and corresponding normal maps, each of the first images includes an image defect, and the second image set includes one or more second images, each of the second images does not include the image defect; and generates one or more third images using the first image set and the second image set, wherein the third images are used to provide condition information associated with the image defect;
[0020] The second module of the image generation system generates one or more fourth images based on a conditional generative adversarial network using the one or more third images, each of which contains one or more image defects.
[0021] The image generation system and image generation method provided in the embodiment of the present application, the first module of the image generation system obtains a first image set and a second image set, the first image set includes one or more first images and corresponding normal maps, each first image includes an image defect, the second image set includes one or more second images, each second image does not include an image defect; and the first image set and the second image set are used to generate one or more third images, and the third images are used to provide conditional information associated with the image defects; the second module of the image generation system is based on a conditional generative adversarial network, and the one or more third images are used to generate one or more fourth images, each fourth image includes one or more image defects. The technical solution provided in the embodiment of the present application, the first module can provide constraints and guidance for the defect image generation process of the second module by using the image after the defect image and the normal image are fused as conditional information (which can also be understood as a conditional variable), so that the second module can learn the characteristics of different defects (such as defect location, size and type), thereby generating a more diverse and high-quality defect image set. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 This is a schematic diagram of the structure of the first image generation system according to an embodiment of the present application;
[0023] Figure 2 This is a schematic diagram of the structure of a second image generation system according to an embodiment of the present application;
[0024] Figure 3 This is an architecture diagram of the defect image synthesis system in the application example of this application;
[0025] Figure 4 This is a schematic diagram of the internal structure of the fusion module in the application example of this application;
[0026] Figure 5 This is a schematic diagram of the internal structure of the conditional generative adversarial network (CGAN) module in the application example of this application;
[0027] Figure 6 This is a schematic diagram of the internal structure of the verification module in the application example of this application;
[0028] Figure 7 The present invention is a flowchart of a method for generating an image according to an embodiment of the present application. DETAILED DESCRIPTION
[0029] The present application is further described in detail below through the accompanying drawings and specific embodiments.
[0030] In the related art, industrial defect image data expansion mainly includes two types of solutions. One solution is to directly transform the original image to generate a new image, thereby increasing the number of images in the defect image data set. The transformation methods include image rotation, image shearing, left-right flipping, random blocking of some areas, adjusting image contrast, adjusting image clarity, etc. The other solution is based on deep learning (such as adversarial generative networks), which automatically generates diverse and more realistic new defect images by learning the intrinsic characteristics and distribution of the original image.
[0031] However, the scheme based on image transformation is only an algebraic operation at the pixel level, which makes it difficult to generate a large number of defect images with different characteristics. Therefore, there is a problem of insufficient diversity of defect images. At the same time, since the shape and characteristics of the defects may change during the transformation process, the generated images cannot accurately reflect the actual defect situation, and the quality of the defect images is not high. In addition, in the scheme based on generative adversarial networks, due to the lack of control over the content of image generation, the training process may be very complex and unstable, making the generated new defect images random and unpredictable.
[0032] In summary, related defect image generation schemes have the problems of insufficient diversity and low image quality.
[0033] Based on this, in various embodiments of the present application, by using the image obtained by fusing the defect image and the normal image as the conditional information of the conditional generative adversarial network, the conditional generative adversarial network can learn the characteristics of different defect types, thereby providing rich contextual information for the generation process of the defect image. In this way, more realistic and diverse defect images can be generated, and at the same time, the stability of the training process can be enhanced.
[0034] The present application embodiment provides an image generation system, such as Figure 1 As shown, it includes: a first module 101 and a second module 102; wherein,
[0035] The first module 101 is used to obtain a first image set and a second image set, wherein the first image set includes one or more first images and corresponding normal maps, each of which includes an image defect, and the second image set includes one or more second images, each of which does not include an image defect; and generate one or more third images using the first image set and the second image set, wherein the third images are used to provide condition information associated with the image defect;
[0036] The second module 102 is configured to generate one or more fourth images based on a conditional generative adversarial network using the one or more third images, each of the fourth images containing one or more image defects.
[0037] Among them, in actual application, the image generation system can be called a defect image synthesis system or an industrial visual defect image synthesis system, etc., which can be applied to the scene of industrial visual defect detection, and at least includes the first module 101 and the second module 102. The first module 101 can be called a fusion module, which at least has an image fusion function. The embodiment of the present application does not limit the name of the first module 101, as long as its function is realized; the second module 102 can be called a generation module or a conditional generative adversarial network module, which can at least generate defect images based on a conditional generative adversarial network. The embodiment of the present application does not limit the name of the second module 102, as long as its function is realized.
[0038] In actual application, for the first image set, the first module 101 can obtain the one or more first images and the corresponding normal map from a pre-configured first image library. The first image library can be called a defect image library. The first image library contains one or more pre-configured defect images and a normal map corresponding to each defect image. The normal map represents the texture features of the image defects in the defect image. The normal map can be stored in an image format, such as a standard RGB image format. In addition, each first image can contain an image defect. The types of image defects can include cracks, holes, scratches, corrosion, etc. The embodiment of the present application does not limit the types of image defects.
[0039] In actual application, for the second image set, the first module 101 can obtain the one or more second images from a pre-configured second image library. The second image library can be called a background image library. The second image library contains one or more pre-configured object images without image defects. These object images can be obtained under different shooting angles, lighting conditions, object postures, resolutions, etc.; in addition, each second image does not contain image defects, so the second image can be called a defect-free image, a normal image or a background image, etc. The embodiment of the present application does not limit the name of the second image.
[0040] Then, the first module 101 can fuse the first image set and the second image set (which can also be understood as superposition processing) to generate one or more third images, each of which can contain one or more image defects, and the third image can be understood as a defective image generated by the first module 101.
[0041] Specifically, in one embodiment, the first module 101 is used to select K first images and normal maps corresponding to the K first images from the first image set, and select a second image from the second image set, where K is an integer greater than or equal to 1; and fuse the K first images, the normal maps corresponding to the K first images, and the selected second image to obtain a third image.
[0042] In actual application, the first module 101 may select K first images (which may be expressed as I d K ) and the corresponding normal map (which can be expressed as N d K ), and at the same time, a second image is selected in a random manner; wherein the value of K can be set as needed, such as pre-configured by the user, and the embodiment of the present application is not limited to this.
[0043] Here, in the process of fusing the K first images, the normal maps corresponding to the K first images and the selected second image, the first module 101 can determine the lighting effect of the normal map, and superimpose the lighting effect of the normal map and the K first images on the selected second image.
[0044] Specifically, in one embodiment, the first module 101 is used to calculate the light intensity characteristics of the K first images using the normal maps corresponding to the K first images; and to fuse the K first images, the light intensity characteristics of the K first images and the selected second image to obtain a third image.
[0045] In actual application, for each first image in the K first images, the first module 101 can use the normal map of the first image to calculate the light intensity characteristics of each pixel in the first image; in this process, the first module can convert the normal map of the first image into a normal vector to calculate the light intensity characteristics of the first image through the normal vector.
[0046] Specifically, in one embodiment, the first module 101 is used to, for each of the K first images, sequentially convert each pixel in the normal map corresponding to the first image to obtain a normal vector corresponding to the first image; and calculate the light intensity characteristics of the first image using the normal vector corresponding to the first image.
[0047] In practical application, for each first image, the first module 101 can convert each pixel in the normal map (which can be expressed as (R, G, B)) into a component of a normal vector (with a value range of 0 to 255), and then normalize the components of the normal vector to obtain a normal vector with a value range of [-1, 1]; wherein, for the Kth first image, the following formula can be used for normalization:
[0048]
[0049] Among them, (R K ,G K ,B K ) is represented as a pixel in the normal map corresponding to the Kth first image, Represented as a normal vector corresponding to the Kth first image.
[0050] Here, after obtaining the normal vector corresponding to the first image, the first module 101 may calculate the light intensity feature of each pixel in the first image, thereby obtaining the light intensity feature of the first image.
[0051] Exemplarily, assuming that the light source direction of the Kth first image is L, the first module 101 may calculate the light intensity feature of the pixel based on the light source direction and the normal vector of the pixel; wherein the calculation may be performed using the following formula:
[0052]
[0053] in, It is represented as the illumination intensity feature of a pixel in the Kth first image, L x ,L y ,L z Represented as the light source direction of the Kth first image.
[0054] In actual application, after obtaining the illumination intensity features of K first images, the first module 101 can sequentially superimpose each first image and the illumination intensity feature on the selected second image to realize image fusion processing; specifically, the first module 101 can determine the image size (also understood as the dimension) of the second image, and determine the center position of the second image based on the image size of the second image; then, for each first image, a first random number and a second random number are respectively generated in a random manner, and with the center position of the second image as a reference, a horizontal movement is performed based on the first random number, and a vertical movement is performed based on the second random number to determine the position of the first image on the second image; then, based on the determined position of the first image, the first image and the illumination intensity feature are sequentially superimposed on the second image to obtain a third image.
[0055] For example, assuming that the second image (denoted as ) is The Kth first image (denoted as ) is And the position on the second image is (x K ,y K ), the first module 101 processes the third image (expressed as ) to initialize (which can be expressed as ), based on the position on the second image, the Kth first image is superimposed on the second image, and then the illumination intensity feature of the Kth first image is superimposed on the second image; wherein, the Kth first image can be superimposed by formula (3), and the illumination intensity feature of the Kth first image can be superimposed by formula (4):
[0056]
[0057] Among them, α K It is represented as the fusion weight of the Kth first image, which can be set as needed, [i, j] represents the center position of the second image, and c represents the color channel R, G, B.
[0058]
[0059] Combining formula (3) and formula (4), for each first image in the K first images, the image after being superimposed on the second image can be expressed as:
[0060]
[0061] It should be noted that, in order to enable each first image to be superimposed within the range of the second image, that is, not exceeding the boundary of the second image, the above superposition process also needs to satisfy one or more of the following conditions (which can also be understood as at least one of them):
[0062]
[0063] In actual application, after the first module 101 generates the one or more third images, the second module 102 can input the one or more third images as conditional information (which can also be understood as conditional variables) into the conditional generative adversarial network, and use the conditional generative adversarial network to generate the one or more fourth images to enhance the diversity and authenticity of the image data set. The fourth image can be understood as a forged defect image; in the above process, the one or more third images can provide conditional information associated with image defects (such as one or more of the defect type, location and size), so as to provide guidance and constraints for the generation process of the fourth image.
[0064] It should be noted that the first module 101 may store the generated one or more third images in a third image library, which may be called a fusion image library and may store images generated by the first module 101. The second module 102 may obtain the one or more third images from the third image library so as to subsequently generate a fourth image.
[0065] Here, for the conditional generative adversarial network, the second module 102 can perform adversarial training through real images and generated fourth images, so that the conditional generative adversarial network can learn the characteristics of different image defects during the adversarial training process, thereby generating a more diverse and high-quality fourth image.
[0066] Based on this, in one embodiment, the second module 102 is also used to obtain one or more real images; based on the one or more real images and the one or more fourth images, adjust the network parameters of the conditional generative adversarial network so that the conditional generative adversarial network has a convergence direction.
[0067] Among them, the conditional generative adversarial network can include a generator and a discriminator, the generator is used to generate a fourth image, and the discriminator is used to classify the generated fourth image, so as to subsequently adjust the network parameters of the conditional generative adversarial network (such as generator weights or discriminator weights) based on the classification results.
[0068] In actual application, the second module 102 can obtain the one or more real images from a fourth image library, and the fourth image library can be called a real image library. The fourth image library contains real images collected in advance from a real environment (for example, collected by detection equipment in an industrial environment), and each real image has one or more real image defects and background; wherein the one or more real images can be obtained in a random manner.
[0069] Next, the second module 102 can use the one or more third images and random noise as inputs of the generator to generate the one or more fourth images through the generator. In this way, the generator can explore the latent space based on random noise to generalize to defect types that have not appeared before, thereby improving the diversity and quality of the generated fourth images as a whole.
[0070] Specifically, in one embodiment, the second module 102 is used to obtain one or more noise vectors; based on a conditional generative adversarial network, the one or more third images and the one or more noise vectors are used to generate the one or more fourth images.
[0071] In actual application, since the second module 102 can obtain the one or more noise vectors from a normal distribution in a random manner, the noise vector can also be called a random noise vector.
[0072] Here, during the adversarial training process, the second module 102 may use the fourth image generated by the generator and the acquired real image as inputs of the discriminator, so as to distinguish the generated fourth image through the discriminator.
[0073] Specifically, in one embodiment, the second module 102 is used to classify the one or more fourth images using the one or more real images to obtain a classification result, wherein the classification result characterizes that the one or more fourth images are real images or forged images; and use the classification result to adjust the network parameters of the conditional generative adversarial network.
[0074] In actual application, during adversarial training, the second module 102 may input the one or more third images as conditional information into the discriminator, so that the discriminator can classify the fourth image more accurately.
[0075] Specifically, in one embodiment, the second module 102 is used to classify the one or more fourth images using the one or more real images and the one or more third images to obtain classification results; use the classification results to calculate the loss value of the conditional generative adversarial network, and use the loss value to adjust the network parameters of the conditional generative adversarial network.
[0076] In practical application, in order to measure the ability of the discriminator to distinguish between real images and forged images, the second module 102 can use the classification result to determine the first probability that the discriminator correctly identifies the fourth image as a forged image and the second probability that the fourth image is incorrectly identified as a real image; using the first probability and the second probability, the loss value of the discriminator is calculated, and the network parameters of the discriminator are adjusted based on the loss value of the discriminator to minimize the loss value of the discriminator, so that the conditional generative adversarial network has a convergence direction; wherein the loss value of the discriminator can be calculated by the following formula:
[0077] D Loss =-∑[logD(I real )]-∑[logD(1-(I fake )] (7)
[0078] Among them, D Loss Represented as the loss value of the discriminator, I real Expressed as the first probability, I fake Expressed as the second probability.
[0079] In addition, the second module 102 can also use the classification result to determine the second probability that the discriminator mistakenly identifies the fourth image as a real image; use the second probability to calculate the loss value of the generator to measure the ability of the generator to generate realistic images (that is, the ability to make it difficult for the discriminator to distinguish between real images and forged images) and the ability to generate images based on conditional information (that is, the ability of the generated image to be consistent with the conditional information in content), so as to adjust the network parameters of the generator so that the conditional generative adversarial network has a convergence direction; wherein the loss value of the generator can be calculated by the following formula:
[0080] G Loss =-∑[logD(I fake )] (8)
[0081] Among them, G Loss Represented as the loss value of the generator.
[0082] In actual application, after completing the training of the conditional generative adversarial network, the second module 102 can also verify the validity of the fourth image generated by the generator.
[0083] Based on this, in one embodiment, if Figure 2 As shown, the system may further include: a third module 103; wherein,
[0084] The third module 103 is used to use one or more real images and the one or more fourth images to train the first model and the second model respectively to obtain the trained first model and the second model; use the trained first model and the second model to perform defect detection on one or more test images to obtain the first detection result and the second detection result respectively; and use the first detection result and the second detection result to determine whether the one or more fourth images meet the requirements.
[0085] The first model and the second model can be understood as models for detecting defective images, and the first model can be expressed as M real , the second model can be expressed as M fake .
[0086] In actual application, the third module 103 can obtain the one or more real images from the fourth database, and use the one or more real images to train the first model; at the same time, the third module can obtain the one or more fourth images from the third image library, and use the one or more fourth images to train the second model; wherein, the number of fourth images obtained is the same as the number of real images, and the embodiment of the present application does not limit the number of images obtained.
[0087] Here, after completing the training of the first model and the second model, the third module 103 can obtain one or more real images from the fourth database, and use the obtained real images as test images to perform defect detection using the trained first model and the second model, thereby obtaining the first detection result and the second detection result, wherein the first detection result represents the detection ability of the first model for defective images, and the first detection result can include the F1-score (which can be expressed as F1 in English) of the first model. real -score), the second detection result represents the detection ability of the second model for the defect image, and the second detection result may include the F1-score (which can be expressed as F2 real -score).
[0088] Exemplarily, for the trained first model, the third module 103 can calculate a first ratio of the number of test images correctly identified as real images to the number of all images identified as real images, and at the same time, calculate a second ratio of the number of test images correctly identified as real images to the number of test images; using the first ratio and the second ratio, the F1-score of the first model is obtained. For the trained second model, the third module 103 can calculate a third ratio of the number of test images correctly identified as real images to the number of all images identified as real images, and at the same time, calculate a fourth ratio of the number of test images correctly identified as real images to the number of test images; using the third ratio and the fourth ratio, the F1-score of the second model is obtained.
[0089] In actual application, by comparing the F1-score of the first model with the F1-score of the second model, the third module 103 can determine whether the generated fourth image meets the requirements; when the F1-score of the second model is greater than or equal to the F1-score of the first model, the third module 103 can determine that the generated fourth image meets the requirements. In this case, the second module 102 can continue to generate a new fourth image through the generator; when the F1-score of the second model is less than the F1-score of the first model, the third module 103 can determine that the generated fourth image does not meet the requirements. In this case, the third module 103 can adjust the fusion strategy and / or network parameters of the conditional adversarial generation network of the first module 101 to optimize the generation process of the fourth image until the F1-score of the second model is greater than or equal to the F1-score of the first model.
[0090] Exemplarily, when the F1-score of the second model is smaller than the F1-score of the first model, the third module 103 may adjust the number of first images selected by the first module 101 during the fusion process, that is, the value of K, or the light source direction of the first image.
[0091] The image generation system provided by the embodiment of the present application, the first module of the image generation system obtains a first image set and a second image set, the first image set includes one or more first images and corresponding normal maps, each first image includes an image defect, the second image set includes one or more second images, each second image does not include an image defect; and the first image set and the second image set are used to generate one or more third images, and the third images are used to provide conditional information associated with the image defects; the second module of the image generation system is based on a conditional generative adversarial network, and the one or more third images are used to generate one or more fourth images, each fourth image includes one or more image defects. The technical solution provided by the embodiment of the present application, the first module uses the image after the defect image and the normal image are fused as conditional information (which can also be understood as a conditional variable) to provide constraints and guidance for the defect image generation process of the second module, so that the second module can learn the characteristics of different defect types, thereby generating a more diverse and high-quality defect image set.
[0092] The present application is described in further detail below in conjunction with application examples.
[0093] In the application example of this application, a defect image synthesis system in an industrial visual scene is proposed; specifically, Figure 3 As shown, the defect image synthesis system (i.e., the above-mentioned image generation system) includes a defect image library (i.e., the above-mentioned first image library), a background image library (i.e., the above-mentioned second image library), a real image library (i.e., the above-mentioned fourth image library), a fusion module (i.e., the above-mentioned first module), a CGAN module (i.e., the above-mentioned second module) and a verification module (i.e., the above-mentioned third module); wherein,
[0094] A fusion module, used to fuse the defect image (i.e., the first image) and the normal map in the defect image library with the defect-free image (i.e., the second image) in the background image library to generate a new defective image (i.e., the third image);
[0095] The CGAN module is used to generate new defect images (i.e., the fourth image mentioned above) using a conditional generative adversarial network to further enhance the diversity and authenticity of the dataset;
[0096] The verification module is used to verify the defective images generated by the CGAN module using defective images in the real image library to evaluate whether the generated defective images meet the requirements.
[0097] Specifically, the defect generation process of the defect image synthesis system includes the following steps:
[0098] Step 1: The fusion module obtains the defect image and the normal map of the defect image from the defect image library;
[0099] Step 2: The fusion module obtains the defect-free image from the background image library;
[0100] Step 3: The fusion module superimposes and fuses the acquired defect image, the normal map of the defect image and the defect-free image;
[0101] Here, if Figure 4 As shown in FIG. 1 , the fusion module superimposes the defect image and the normal map of the defect image onto the defect-free image, thereby obtaining a fused image (which can be expressed as I f ), and then, the fused image is stored in a fused image library (ie, the third image library mentioned above).
[0102] Step 4: The CGAN module generates new defective images using synthetic images with defects and noise, and performs adversarial training on a real image library;
[0103] Here, if Figure 5 As shown in the figure, first, the CGAN module initializes the parameters of the generator and the discriminator; secondly, the fused image and the random noise vector (which can be expressed as z) are used as the input of the generator to generate a new defect image through the generator; then, the CGAN module obtains the real image from the real image library, and uses the real image, the new defect image and the fused image as the input of the discriminator to distinguish the real image from the forged image through the discriminator, that is, calculates the output result of the discriminator for the real image (that is, the first probability mentioned above) and the output result of the discriminator for the forged image (that is, the second probability mentioned above).
[0104] Then, the CGAN module calculates the loss value of the discriminator through formula (7) according to the output result of the discriminator, and updates the parameters of the discriminator to minimize the loss value of the discriminator; at the same time, the CGAN module calculates the loss value of the generator through formula (8) according to the output result of the discriminator, and updates the parameters of the generator to minimize the loss value of the generator.
[0105] It should be noted that the CGAN module will continue to perform the above adversarial training process until the conditional adversarial generation network converges. After the training is completed, the generator can generate realistic defect images based on the input fusion image and random noise, while the discriminator can distinguish between real defect images and generated defect images to verify the quality of the generated defect images.
[0106] Step 5: The verification module uses the defect images generated by the CGAN module and the real defect images to train the models respectively, and compares the performance of the trained models; based on the comparison results, the fusion strategy and the network parameters of the CGAN module are adjusted to optimize the generation process.
[0107] Here, the internal implementation process of the verification module is as follows: Figure 6 As shown, the following steps are included:
[0108] Step 1: The verification module obtains defect images from the real image library and divides the obtained defect images into a training set (which can be expressed as ) and the test set (which can be expressed as );
[0109] Step 2: The verification module obtains the same number of images as the training set from the defect images generated by the CGAN module to construct the training set (which can be expressed as )
[0110] Step 3: Verify that the modules are used separately For the first model M real and the second model M fake Perform training to obtain a trained first model and a trained second model;
[0111] Step 4: Verify Module Exploitation Calculate the F1-scores of the first model and the second model respectively;
[0112] Step 5: The verification module determines whether the defect image generated by the CGAN module meets the requirements by comparing the F1-scores of the first model and the second model, so as to verify the validity of the generated defect image; if the requirements are not met, the fusion module or the CGAN module can be adjusted and training can continue.
[0113] In this application example, a conditional adversarial generative network is used to generate defect images, and a complete network structure with the fused defect image as conditional information is proposed; by taking the defect image, the normal map of the defect image, and the fused image of the background image as conditional information (such as defect type, location, size, etc.), additional guidance and constraints are provided for the generation process of the defect image, which can not only enhance the stability of the training process, but also accurately control the type and distribution of defects in the generation stage, thereby helping to generate defect images that are closer to real samples and more diverse.
[0114] Secondly, the CGAN module generates defect images by learning data distribution rather than simple mathematical operations at the pixel level. Therefore, it can generate more realistic defect images. In addition, the CGAN module essentially generates defect images based on random noise. Therefore, it has a great advantage in generating the number of new defect images.
[0115] Based on the image generation system of the embodiment of the present application, the embodiment of the present application also provides an image generation method, such as Figure 7 As shown, the method includes:
[0116] Step 701: The first module of the image generation system acquires a first image set and a second image set, wherein the first image set includes one or more first images and corresponding normal maps, each of which includes an image defect, and the second image set includes one or more second images, each of which does not include an image defect; and generates one or more third images using the first image set and the second image set, wherein the third images are used to provide condition information associated with the image defect;
[0117] Step 702: The second module of the image generation system generates one or more fourth images based on a conditional generative adversarial network using the one or more third images, each of the fourth images containing one or more image defects.
[0118] In one embodiment, the second module of the image generation system generates one or more fourth images based on a conditional generative adversarial network using the one or more third images, including:
[0119] The second module obtains one or more noise vectors;
[0120] The second module generates the one or more fourth images based on a conditional generative adversarial network using the one or more third images and the one or more noise vectors.
[0121] In one embodiment, the second module of the image generation system generates one or more fourth images based on a conditional generative adversarial network using the one or more third images, including:
[0122] The second module acquires one or more real images;
[0123] The second module adjusts network parameters of the conditional generative adversarial network based on the one or more real images and the one or more fourth images so that the conditional generative adversarial network has a convergence direction.
[0124] In one embodiment, the second module adjusts network parameters of the conditional generative adversarial network based on the one or more real images and the one or more fourth images so that the conditional generative adversarial network has a convergence direction, including:
[0125] The second module classifies the one or more fourth images using the one or more real images to obtain a classification result, wherein the classification result indicates that the one or more fourth images are real images or forged images;
[0126] The second module uses the classification result to adjust the network parameters of the conditional generative adversarial network.
[0127] In one embodiment, the second module classifies the one or more fourth images using the one or more real images to obtain a classification result, including:
[0128] The second module classifies the one or more fourth images using the one or more real images and the one or more third images to obtain a classification result;
[0129] The second module uses the classification result to adjust the network parameters of the conditional generative adversarial network, including:
[0130] The second module uses the classification result to calculate the loss value of the conditional generative adversarial network, and uses the loss value to adjust the network parameters of the conditional generative adversarial network.
[0131] In one embodiment, the step of generating one or more third images by using the first image set and the second image set includes:
[0132] The first module selects K first images and normal maps corresponding to the K first images from the first image set, and selects a second image from the second image set, where K is an integer greater than or equal to 1;
[0133] The first module performs fusion processing on the K first images, the normal maps corresponding to the K first images and the selected second image to obtain a third image.
[0134] In one embodiment, the first module performs fusion processing on the K first images, the normal maps corresponding to the K first images, and the selected second image to obtain a third image, including:
[0135] The first module calculates the illumination intensity characteristics of the K first images using the normal maps corresponding to the K first images;
[0136] The first module performs fusion processing on the K first images, the illumination intensity features of the K first images and the selected second image to obtain a third image.
[0137] In one embodiment, the first module calculates the illumination intensity features of the K first images using the normal maps corresponding to the K first images, including:
[0138] The first module converts each pixel in the normal map corresponding to each first image of the K first images in turn to obtain a normal vector corresponding to the first image; and uses the normal vector corresponding to the first image to calculate the illumination intensity characteristics of the first image.
[0139] In one embodiment, the method may further include:
[0140] The third module of the image generation system uses the one or more real images and the one or more fourth images to train the first model and the second model respectively, to obtain the trained first model and the second model;
[0141] The third module uses the trained first model and the second model to perform defect detection on one or more test images to obtain a first detection result and a second detection result respectively;
[0142] The third module determines whether the one or more fourth images meet the requirements by using the first detection result and the second detection result.
[0143] It should be noted that: "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0144] In addition, the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.
[0145] The above is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the scope of protection of the present application.
Claims
1. An image generation system, characterized in that: include: The first module and the second module; wherein, The first module is used to obtain a first image set and a second image set, wherein the first image set includes one or more first images and corresponding normal maps, each of which includes an image defect, and the second image set includes one or more second images, each of which does not include an image defect; and generate one or more third images using the first image set and the second image set, wherein the third images are used to provide condition information associated with the image defect; The second module is used to generate one or more fourth images based on a conditional generative adversarial network using the one or more third images, each of the fourth images containing one or more image defects.
2. The system according to claim 1, characterized in that The second module is used to obtain one or more noise vectors; based on a conditional generative adversarial network, the one or more third images and the one or more noise vectors are used to generate the one or more fourth images.
3. The system according to claim 1, characterized in that The second module is also used to obtain one or more real images; based on the one or more real images and the one or more fourth images, adjust the network parameters of the conditional generative adversarial network so that the conditional generative adversarial network has a convergence direction.
4. The system according to claim 3, characterized in that The second module is used to classify the one or more fourth images using the one or more real images to obtain a classification result, wherein the classification result indicates whether the one or more fourth images are real images or forged images; Using the classification results, the network parameters of the conditional generative adversarial network are adjusted.
5. The method according to claim 4, characterized in that The second module is used to classify the one or more fourth images using the one or more real images and the one or more third images to obtain a classification result; Using the classification results, a loss value of the conditional generative adversarial network is calculated, and the loss value is used to adjust the network parameters of the conditional generative adversarial network.
6. The system according to claim 1, characterized in that The first module is used to select K first images and normal maps corresponding to the K first images from the first image set, and select a second image from the second image set, where K is an integer greater than or equal to 1; and fuse the K first images, the normal maps corresponding to the K first images, and the selected second image to obtain a third image.
7. The system according to claim 6, characterized in that The first module is used to calculate the illumination intensity features of the K first images using the normal maps corresponding to the K first images; and to fuse the K first images, the illumination intensity features of the K first images and the selected second image to obtain a third image.
8. The system according to claim 7, characterized in that The first module is used to convert each pixel in the normal map corresponding to each first image of the K first images in turn to obtain the normal vector corresponding to the first image; and use the normal vector corresponding to the first image to calculate the light intensity characteristics of the first image.
9. The system according to any one of claims 1 to 8, characterized in that: The system further comprises: a third module; wherein, The third module is used to use one or more real images and the one or more fourth images to train the first model and the second model respectively, to obtain the trained first model and the second model; use the trained first model and the second model to perform defect detection on one or more test images, to obtain the first detection result and the second detection result respectively; use the first detection result and the second detection result to determine whether the one or more fourth images meet the requirements.
10. An image generation method, characterized in that: Applied to image generation systems, including: The first module of the image generation system acquires a first image set and a second image set, wherein the first image set includes one or more first images and corresponding normal maps, each of the first images includes an image defect, and the second image set includes one or more second images, each of the second images does not include the image defect; and generates one or more third images using the first image set and the second image set, wherein the third images are used to provide condition information associated with the image defect; The second module of the image generation system generates one or more fourth images based on a conditional generative adversarial network using the one or more third images, each of which contains one or more image defects.