Defective zero sample generation method, device and equipment based on image generation method

Abstract

The present invention provides a defect zero sample generation method, device and equipment based on an image generation method, including: obtaining a defect-free image corresponding to a first scene and a defect image corresponding to a second scene; identifying defects contained in the defect image through a pre-trained defect image cutout model to generate a defect mask map; synthesizing the defect-free image and the defect mask map to obtain an initial defect image corresponding to the first scene; adjusting the foreground or background in the initial defect image through a pre-trained image harmonization model so that the foreground of the adjusted initial defect image is harmonious with its background, and obtaining a target defect image corresponding to the first scene. The present invention can provide a large number of defect images for defect detection based on deep learning.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device and equipment for generating a defect zero sample based on an image generation method. Background Art

[0002] Defect detection has very important applications in the industrial field. Among them, the traditional detection algorithm based on defect detection is not effective in practice. The reason why the deep learning method cannot be directly applied to surface defect detection is that in the real industrial environment, there are too few industrial defect samples that can be provided. It takes a huge cost to obtain a large number of samples for training, and some dangerous samples are even impossible to obtain. Summary of the invention

[0003] In view of this, an object of the present invention is to provide a method, device and equipment for generating zero defect samples based on an image generation method, which can provide a large number of defect images for defect detection based on deep learning.

[0004] In a first aspect, an embodiment of the present invention provides a method for generating a defect zero sample based on an image generation method, comprising:

[0005] Acquire a non-defective image corresponding to the first scene and a defective image corresponding to the second scene;

[0006] Using a pre-trained defect image cutout model, the defects contained in the defect image are identified to generate a defect mask map;

[0007] synthesizing the defect-free image and the defect mask map to obtain an initial defect image corresponding to the first scene;

[0008] The foreground or background in the initial defect image is adjusted through a pre-trained image harmonization model so that the foreground of the adjusted initial defect image is harmonious with its background, thereby obtaining a target defect image corresponding to the first scene.

[0009] In one embodiment, the defect image matting model includes a shared encoder, a pyramid pooling unit, a semantic context branch, and an improved high-resolution detail branch;

[0010] The step of identifying defects contained in the defect image by using a pre-trained defect image cutout model to generate a defect mask map comprises:

[0011] Extracting multi-scale defect feature information corresponding to the defect image through the shared encoder;

[0012] The defect feature information of the first scale is pooled by the pyramid pooling unit, and the pooled defect feature information is subjected to global context extraction by the semantic context branch to obtain multi-scale context feature information and a semantically-aware trimap.

[0013] Generate a defect detail map based on the defect feature information at the second scale, the defect feature information at the third scale, and the context feature information through the improved high-resolution detail branch;

[0014] The semantically-aware trimap image and the defect detail image are fused to obtain a defect mask map.

[0015] In one embodiment, the improved high-resolution detail branch includes a hybrid attention unit, a multi-level residual block, and a convolutional layer;

[0016] The step of generating a defect detail map based on the defect feature information at the second scale, the defect feature information at the third scale and the context feature information through the improved high-resolution detail branch comprises:

[0017] performing upsampling processing on the defect feature information at the second scale and the defect feature information at the third scale respectively;

[0018] By means of the hybrid attention unit, feature extraction and fusion processing are performed on the upsampled defect feature information based on the hybrid attention mechanism to obtain fused feature information;

[0019] The fused feature information and the context feature information are processed by multiple levels of residual blocks and convolutional layers to generate a defect detail map.

[0020] In one embodiment, the hybrid attention unit includes a channel attention subunit and a spatial attention subunit;

[0021] The step of extracting and fusing the upsampled defect feature information based on the hybrid attention mechanism through the hybrid attention unit to obtain the fused feature information includes:

[0022] By means of the channel attention subunit, feature extraction is performed on the upsampled defect feature information based on the channel attention mechanism to obtain first feature information;

[0023] By means of the spatial attention subunit, feature extraction is performed on the upsampled defect feature information based on the spatial attention mechanism to obtain second feature information;

[0024] An element-wise addition operation is performed on the first feature information and the second feature information to obtain fused feature information.

[0025] In one embodiment, the step of generating a defect detail map after the fused feature information and the context feature information are processed by multiple levels of residual blocks and convolutional layers includes:

[0026] For the residual block located at the first position, inputting the fused feature information into the residual block, so that the residual block outputs defect detail features;

[0027] For the other residual blocks, upsampling the context feature information so that the scale of the upsampled context feature information is the same as that of the defect detail feature output by the previous residual block, and fusion of the upsampled context feature information and the defect detail feature output by the previous residual block is performed, and the fused feature is input to the residual block so that the residual block outputs the defect detail feature;

[0028] The defect detail features output by the last residual block and the context feature information of the same scale are fused, and the fused features are input into the convolution layer, so that the convolution layer outputs a defect detail map.

[0029] In one embodiment, the step of synthesizing the defect-free image and the defect mask map to obtain an initial defect image corresponding to the first scene includes:

[0030] Performing target recognition on the defect-free image to determine the location of the target object in the defect-free image;

[0031] Based on a random scaling ratio and / or rotation angle, performing image processing on the defect mask map to obtain a processed defect mask map;

[0032] Based on the position of the target object, the defect-free image and the processed defect mask map are synthesized to obtain an initial defect image corresponding to the first scene.

[0033] In one embodiment, the image harmonization model includes a generator, a global discriminator, and a verification discriminator;

[0034] The training steps of the image harmonization model include:

[0035] Generate a second defect sample based on the synthesized first defect sample and its corresponding defect mask sample by the generator;

[0036] Performing global identification on the real image corresponding to the first defect sample and the second defect sample by the global identifier to generate a global identification result;

[0037] And, by means of the verification and identification device, the real image corresponding to the first defect sample and the second defect sample are verified and identified to generate a verification and identification result;

[0038] The generator, the global discriminator and the verification discriminator are trained using the global identification result and the verification identification result; wherein the trained generator is used to adjust the foreground or background in the initial defect image to obtain a target defect image corresponding to the first scene.

[0039] In a second aspect, an embodiment of the present invention further provides a defect zero sample generating device based on an image generating method, comprising:

[0040] An image acquisition module, used to acquire a non-defective image corresponding to the first scene and a defective image corresponding to the second scene;

[0041] A mask generation module, used to identify defects contained in the defect image through a pre-trained defect image matting model to generate a defect mask map;

[0042] An image synthesis module, used for synthesizing the defect-free image and the defect mask map to obtain an initial defect image corresponding to the first scene;

[0043] The image harmonization module is used to adjust the foreground or background in the initial defect image through a pre-trained image harmonization model so that the foreground of the adjusted initial defect image is harmonious with its background, thereby obtaining a target defect image corresponding to the first scene.

[0044] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement any one of the methods provided in the first aspect.

[0045] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement any one of the methods provided in the first aspect.

[0046] The embodiment of the present invention provides a defect zero-sample generation method, device and equipment based on an image generation method. First, a defect-free image corresponding to a first scene and a defect image corresponding to a second scene are obtained; then, a defect-free image corresponding to a first scene and a defect image corresponding to a second scene are identified through a pre-trained defect image cutout model to generate a defect mask map; then, the defect-free image and the defect mask map are synthesized to obtain an initial defect image corresponding to the first scene; finally, the foreground or background in the initial defect image is adjusted through a pre-trained image harmonization model so that the foreground of the adjusted initial defect image is harmonious with its background, and a target defect image corresponding to the first scene is obtained. The above method uses a defect image cutout model to generate a corresponding defect mask map based on the defect image corresponding to the second scene. After synthesizing the defect-free image and the defect mask map, the image harmonization model is used to adjust the foreground or background of the synthesized initial defect image to make it harmonious with the background to obtain a target defect image with a better effect. The embodiment of the present invention uses image cutout, image synthesis and harmonization technology to generate defect images in the case of zero samples, providing a large number of defect images with better effects for defect detection based on deep learning.

[0047] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0048] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 A schematic diagram of a flow chart of a method for generating a defect zero sample based on an image generation method provided by an embodiment of the present invention;

[0051] Figure 2 An overall flow chart of a defect zero sample generation method based on an image generation method provided by an embodiment of the present invention;

[0052] Figure 3 A framework diagram of a defect image cutout model provided by an embodiment of the present invention;

[0053] Figure 4 A schematic diagram of the specific structure of a defect image cutout model provided by an embodiment of the present invention;

[0054] Figure 5 A schematic diagram of the structure of a hybrid attention unit provided by an embodiment of the present invention;

[0055] Figure 6 A schematic diagram of a defect-free image and an initial defective image provided by an embodiment of the present invention;

[0056] Figure 7 A framework diagram of an image harmonization model provided by an embodiment of the present invention;

[0057] Figure 8 A structural schematic diagram of an image harmonization model provided by an embodiment of the present invention;

[0058] Fig. 9 A schematic structural diagram of a defect zero sample generating device based on an image generating method provided by an embodiment of the present invention;

[0059] Fig.10 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described in combination with the embodiments below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] At present, due to the lack of industrial defect samples, deep learning-based methods cannot be directly applied to surface defect detection. Based on this, the present invention implements a method, device and equipment for generating zero defect samples based on image generation, which can provide a large number of defect images for deep learning-based defect detection.

[0062] To facilitate understanding of this embodiment, firstly, a defect zero sample generation method based on an image generation method disclosed in an embodiment of the present invention is described in detail, see Figure 1 The flowchart of a method for generating a defect zero sample based on an image generation method is shown in FIG. 1 , and the method mainly includes the following steps S102 to S108:

[0063] Step S102 , obtaining a non-defective image corresponding to the first scene and a defective image corresponding to the second scene.

[0064] The first scene may be a scene lacking defective images, and the second scene may be a scene with existing defective images. For example, assuming that when performing defect detection on an underground pipe gallery, there is a lack of images of the underground pipe gallery with defects, then the scene of performing defect detection on the underground pipe gallery is the first scene; if there is an image of a steel plate with defects, it can be determined as the second scene, and the corresponding defective image can be collected.

[0065] Step S104, identifying defects contained in the defect image by using a pre-trained defect image cutout model to generate a defect mask map.

[0066] Among them, the defect image matting model can adopt the PP-Matting network architecture, and add a hybrid attention unit on its basis. The defect image matting model is used to identify the defects contained in the defect image, and a high-precision defect mask map can be generated.

[0067] Step S106, synthesizing the defect-free image and the defect mask map to obtain an initial defect image corresponding to the first scene.

[0068] In one embodiment, the defect-free image and the defect mask map can be synthesized automatically or manually. Taking the automatic method as an example, the target object in the defect-free image can be identified, such as the location of the underground pipe gallery in the defect-free image, and the defect mask map is scaled and / or rotated, and then the scaled and / or rotated defect mask map and the defect-free image are synthesized based on the location of the underground pipe gallery to generate an initial defect image, in which the underground pipe gallery in the initial defect image has a defect.

[0069] Step S108, adjusting the foreground or background in the initial defect image through a pre-trained image harmonization model, so that the foreground of the adjusted initial defect image is harmonized with its background, and obtaining a target defect image corresponding to the first scene.

[0070] Considering that the foreground and background parts of the synthesized initial defect image are taken under different shooting conditions (such as time, season, lighting, and weather), there are obvious mismatches in brightness and color. Therefore, the initial defect image is input into the image harmonization model, and the foreground and / or background parts thereof are adjusted using the image harmonization model to make the foreground and background parts harmonious, thereby obtaining the target defect image corresponding to the first scene. Optionally, the image harmonization model adopts an adversarial generative network, and by inputting the initial defect image and the defect mask map into the generator in the adversarial generative network, the generator can output the target defect image.

[0071] The defect zero-sample generation method based on image generation provided by the embodiment of the present invention utilizes a defect image cutout model to generate a corresponding defect mask map based on the defect image corresponding to the second scene, and after synthesizing the defect-free image and the defect mask map, uses the image harmonization model to adjust the foreground or background of the synthesized initial defect image to make it harmonious with the background, so as to obtain a target defect image with better effect. The embodiment of the present invention utilizes image cutout, image synthesis and harmonization technology to generate defect images in the case of zero samples, thereby providing a large number of defect images with better effects for defect detection based on deep learning.

[0072] To facilitate understanding of the above embodiments, the present invention provides a specific implementation of a method for generating a defect zero sample based on an image generation method, see Figure 2 The overall flow chart of a defect zero-sample generation method based on an image generation method is shown, comprising: obtaining defect images corresponding to other existing scenes (i.e., the aforementioned second scene); generating a defect mask map (i.e., a defect mask map) through a defect image cutout model based on an attention mechanism; harmonizing the defect mask map and the defect-free image of the defect-free scene (i.e., the aforementioned first scene), and outputting a target defect image of the defect-free scene.

[0073] First, the defect image cutout model is explained. Figure 3 The framework diagram of a defect image matting model shown in the figure is that the defect image of the second scene is input into the defect image matting model, and the corresponding defect mask map can be output. Optionally, the defect image matting model adopts the PP-Matting network architecture, and adds a hybrid attention unit on its basis. Specifically, the defect image matting model includes a shared encoder, a pyramid pooling (PPM) unit, a semantic context branch (SCB), and an improved high-resolution detail branch (HRDB). Among them, considering the powerful ability of the HRNet network to extract high-level semantic information, the shared encoder adopts the HRNet network and uses it as the backbone network. A series of parameter adjustments are made on the backbone to obtain a larger receptive field. The defect image matting model also designs two branches, SCB and HRDB, to perform semantic and detail predictions respectively. Through the guided flow mechanism, high-resolution detail predictions under semantic guidance are performed, thereby realizing Trimap-free high-precision image matting.

[0074] On this basis, the embodiment of the present invention provides a specific implementation method for generating a defect mask map, see the following steps 1 to 4:

[0075] Step 1: Extract multi-scale defect feature information corresponding to the defect image through a shared encoder.

[0076] Efficient feature extraction is a common component of any deep learning method. The cutout task requires combining high-level abstract semantic segmentation with high-resolution detail regression. Therefore, in addition to low-resolution feature representation, image cutout also requires high-resolution feature representation. Since HRNet48 has a strong ability in high-resolution feather extraction, the embodiment of the present invention adopts HRNet48 pre-trained on ImageNet as a shared encoder. Instead of restoring high resolution by upsampling, the shared encoder maintains high-resolution representation throughout the process and repeatedly fuses multi-resolution representations to generate rich high-resolution features with strong position sensitivity. The shared encoder is shared between SCB and HRDB.

[0077] In one example, see Figure 4 A specific structural diagram of a defect image cutout model is shown. After the defect image is input into the shared encoder, the shared encoder will output multi-scale defect feature information, including defect feature information with 1 / 2 resolution, defect feature information with 1 / 4 resolution, and defect feature information with 1 / 32 resolution.

[0078] In step 2, the defect feature information of the first scale is pooled through the pyramid pooling unit (PPM), and the global context is extracted from the pooled defect feature information through the semantic context branch (SCB) to obtain multi-scale context feature information and semantic-aware trimap.

[0079] The defect feature information at the first scale is also the defect feature information at a resolution of 1 / 32.

[0080] In one example, the pyramid pooling unit is used to further strengthen the semantic context between the shared encoder and the semantic context branch. Since high-level abstract features are important for correct semantic graphs, the embodiment of the present invention uses the defect feature information with a resolution of 1 / 32 output by the shared encoder as the input of the semantic context branch after being processed by the pyramid pooling unit.

[0081] In practical applications, without the trimap clues, previous cutout methods are difficult to obtain the correct semantic context feature information and are prone to local ambiguity. In this work, the embodiment of the present invention uses a separate semantic context branch for global context extraction. Specifically, the semantic context branch consists of five blocks, each of which consists of three ConvBNReLU (convolution, batch normalization (BN) and rectified linear unit (ReLU)) and one bilinear upsampling. Please continue to refer to Figure 4, the first block will output context feature information with a resolution of 1 / 16, the second block will output context feature information with a resolution of 1 / 8, the third block will output context feature information with a resolution of 1 / 4, the fourth block will output context feature information with a resolution of 1 / 2, and the fifth block will output context feature information with a resolution of 1 / 1.

[0082] Furthermore, the output of the semantic context branch is supervised by the semantic trimap and the semantic map of the semantic context branch is very similar to the trimap map provided by the user, which shows that the semantic context branch is able to automatically generate reliable semantically-aware trimap maps.

[0083] Step 3: Generate a defect detail map based on the defect feature information at the second scale, the defect feature information at the third scale, and the context feature information through an improved high-resolution detail branch (HRDB).

[0084] The degree of fine-grained details is a major difference between matting and segmentation. Detail representations are usually retained in high-resolution feature maps. However, existing encoder-decoder segmentation networks with downsampling-upsampling architectures cannot effectively maintain high-resolution representations. In this work, an embodiment of the present invention uses a high-resolution detail branch to obtain high-resolution fine-grained details. Specifically, the high-resolution detail branch consists of three residual blocks and a convolutional layer. The initial input of the high-resolution detail branch is a combination of two intermediate features with corresponding upsampling scales, which respectively contain low-level texture information (i.e., defect feature information at 1 / 2 resolution) and high-level abstract information (i.e., defect feature information at 1 / 4 resolution).

[0085] Please continue to see Figure 4 , the improved high-resolution detail branch includes a mixed attention unit (MA, MixAttention), a multi-level residual block (Residual Block) and a convolutional layer (Conv). Based on this, the aforementioned step 3 can specifically include the following steps 3.1 to 3.3:

[0086] Step 3.1, respectively upsample the defect feature information of the second scale and the defect feature information of the third scale. Figure 4 , respectively up-sampling the defect feature information with 1 / 2 resolution and up-sampling the defect feature information with 1 / 4 resolution.

[0087] In step 3.2, the upsampled defect feature information is extracted and fused based on the hybrid attention mechanism through the hybrid attention unit to obtain the fused feature information.

[0088] One of the core points of the embodiment of the present invention is to use a hybrid attention unit to process the upsampled defect feature information. Figure 5 The structure diagram of a hybrid attention unit shown in FIG. 4 includes a channel attention subunit and a spatial attention subunit. The spatial attention subunit aims to incorporate more extensive contextual information into local features and enhance their representation capabilities. On the other hand, the channel attention subunit exploits the interdependence between channel maps, highlights the interdependence of feature maps, and enhances the representation of specific semantics. The outputs of the two attention subunits are transformed through a convolutional layer, and feature fusion is achieved using an element-wise addition operation to obtain fused feature information.

[0089] In a specific implementation, the method includes: (1) extracting the upsampled defect feature information based on the channel attention mechanism through a channel attention subunit to obtain first feature information; (2) extracting the upsampled defect feature information based on the spatial attention mechanism through a spatial attention subunit to obtain second feature information; (3) performing element-wise addition operation on the first feature information and the second feature information to obtain fused feature information.

[0090] Please continue to see Figure 5 , Figure 5 X and Y in the figure refer to the defect feature information of 1 / 2 resolution and 1 / 4 resolution after upsampling, respectively. Taking the channel attention subunit as an example, the maximum feature information MAX_X and the average feature information MEAN_X of X are extracted. Similarly, the maximum feature information MAX_Y and the average feature information MEAN_Y of Y are extracted. After CONCAT, Conv and ReLU processing, the first feature information is output; similarly, the spatial attention subunit will output the second feature information. The two feature information are element-wise added to obtain the fused feature information.

[0091] In step 3.3, the fused feature information and context feature information are processed by multi-level residual blocks and convolutional layers to generate a defect detail map.

[0092] In the specific implementation, it includes:

[0093] (a) For the residual block at the first position, the fused feature information is input into the residual block so that the residual block outputs the defect detail features. Figure 4 , the fused feature information is input into the first residual block, and the first residual block will output the corresponding defect detail features.

[0094] (b) For other residual blocks, the context feature information is upsampled so that the scale of the upsampled context feature information is the same as that of the defect detail feature output by the previous residual block, and feature fusion is performed on the upsampled context feature information and the defect detail feature output by the previous residual block. The fused feature is input into the residual block so that the residual block outputs the defect detail feature.

[0095] Please continue to see Figure 4 After upsampling, the context feature information with a resolution of 1 / 16 is input to the second residual block through the guidance flow together with the defect detail feature information output by the first residual block, and the second residual block will output the corresponding defect detail features; after upsampling, the context feature information with a resolution of 1 / 4 is input to the third residual block through the guidance flow together with the defect detail feature information output by the second residual block, and the third residual block will output the corresponding defect detail features.

[0096] Among them, between the two residual blocks, the guided flow is used to obtain the semantic context from the semantic context branch. As discussed above, the global semantic context provides meaningful guidance for detail extraction. In this work, the embodiment of the present invention applies the guided flow strategy of the context feature information output by the guided semantic context branch to assist the detail prediction in the high-resolution detail branch. Specifically, it consists of a gated convolutional layer (GCL) that realizes the interaction between defect detail features and context feature information.

[0097] (c) The defect detail features output by the last residual block and the context feature information of the same scale are fused, and the fused features are input into the convolution layer so that the convolution layer outputs a defect detail map.

[0098] Please continue to see Figure 4 ,The defect detail features output by the third residual block and the ,context feature information with 1 / 1 resolution are input to the convolution layer through the ,guided flow to obtain the defect detail map.

[0099] The output of the high-resolution detail branch in the embodiment of the present invention is a defect detail (alpha) map, which focuses on detail representation in the transition region and is then fused with the semantic map in the semantic context branch to generate the final defect detail map.

[0100] Step 4: fuse the semantically-aware trimap image and the defect detail image to obtain a defect mask map.

[0101] After the PP-Matting network, the defect mask map of the defect image can be extracted. Since the defect images have similar texture features, but the background is inconsistent due to different environments, the scratch type extracted from the steel plate defect sample can be used to synthesize the scratch defect samples in the underground pipeline gallery.

[0102] Furthermore, an embodiment of the present invention also provides a loss function used in training a defect image cutout model, specifically:

[0103] The first loss function is the semantic context branch represented as L s The semantic loss, which is the cross entropy loss for the 3-class segmentation task, is shown below.

[0104]

[0105] Among them, c∈{1,2,3} represents the three classes in the semantic-aware trimap graph. is the predicted probability of the cth class at each i-th pixel, is the corresponding true value, and Ω represents all pixels in the semantically aware trimap.

[0106] The second loss is the detail loss in the high-resolution detail branch, denoted as L d The embodiment of the present invention sums the alpha prediction loss Lα and the gradient loss Lgrad as the detail loss. Since the high-resolution detail branch focuses on detail prediction, the embodiment of the present invention only calculates the detail loss in the transition region:

[0107]

[0108] in, is the alpha prediction loss, is the gradient loss.

[0109] The third loss is the fusion loss in the final alpha mask, denoted as L f , which consists of alpha prediction loss, gradient loss and component loss:

[0110]

[0111] in, The absolute difference between the true RGB color and the predicted RGB color composited with the true foreground, background, and predicted alpha mask.

[0112] Finally, the above three types of losses are weightedly summed to obtain the target loss value, which is used to train the defect image cutout model.

[0113] Furthermore, the embodiment of the present invention also provides an implementation method for synthesizing an initial defect image, see the following steps a to c:

[0114] Step a, target recognition is performed on the defect-free image to determine the location of the target object in the defect-free image. Taking the underground pipeline corridor as an example, the location of the underground pipeline corridor in the defect-free image can be identified by various methods such as image processing or neural network.

[0115] Step b, performing image processing on the defect mask map based on a random scaling ratio and / or rotation angle to obtain a processed defect mask map;

[0116] Step c: based on the position of the target object, synthesize the defect-free image and the processed defect mask map to obtain an initial defect image corresponding to the first scene.

[0117] For example, see Figure 6 A schematic diagram of a defect-free image and an initial defect image is shown. After the defect mask map is determined, the scale and rotation of the defect mask map can be adjusted by image processing, and the position where the defect mask map is to be pasted in the defect-free image can be specified to synthesize the initial defect image.

[0118] Further, the embodiment of the present invention explains the image harmonization model, which includes a generator, a global discriminator and a verification discriminator. In practical applications, the problem of disharmony between foreground and background in the synthetic image can be solved based on the DoveNet network. Figure 7 The framework diagram of an image harmonization model shown in the figure can output a harmonized target defect image by inputting an initial defect image into the image harmonization model. In an embodiment of the present invention, image harmonization aims to adjust the foreground in the composite image to make it harmonious with the background. After passing through the DoveNet network, the purpose of the embodiment of the present invention is to generate a target defect image that is consistent with the background image, so that the target defect image is closer to the actual situation and facilitates the training of the subsequent defect detection network, which is used to detect defects in the first scene.

[0119] Specifically, in the composite image, the foreground and background are taken under different shooting conditions (such as time, season, lighting, weather), so there is an obvious mismatch in brightness and color. Figure 8The figure shows a schematic diagram of the structure of an image harmonization model. The overall structure uses the connection of three models: (a) Attention Enhanced Generator (U-Net adds an attention module), that is, the generator; (b) Global Discriminator (a Discriminator similar to GAN), that is, the global discriminator; (c) Domain Verification Discriminator (a Discriminator for a domain, this domain refers to lighting, weather, season, etc.), that is, the verification discriminator.

[0120] On this basis, the embodiment of the present invention further provides an implementation method for training an image harmonization model, see the following steps 1 to 4:

[0121] Step one, through the generator, based on the synthesized first defect sample and its corresponding defect mask sample, a second defect sample is generated. The generator G provided in the embodiment of the present invention is based on U-Net and has a skip link from the encoder to the decoder. The embodiment of the present invention uses an attention block to enhance U-Net. Specifically, the encoder and decoder features are first connected, and on this basis, the full attention (integration of spatial attention and channel attention) of the encoder features and the decoder features are learned respectively. Then, the participating encoder and decoder features are connected. In total, three attention blocks are inserted into the U-Net in the embodiment of the present invention. In addition, the second defect sample (or target defect image) generated by the absolute error loss is forced to be close to the real image.

[0122] Step 2: Globally discriminate the real image corresponding to the first defect sample and the second defect sample through a global discriminator to generate a global discriminant result. The global discriminator Dg is intended to help G generate plausible images. It regards I as a real image and ^I as a false image. The embodiment of the present invention applies spectral normalization after each convolution layer and uses hinge loss for stable training.

[0123] Step three, through the verification discriminator, the real image corresponding to the first defect sample and the second defect sample are verified and discriminated to generate a verification and discriminant result. In addition to the global discriminator, the embodiment of the present invention also designs a main verification discriminator to verify whether the foreground and background of a given image belong to the same domain. The foreground and background of the real (same, synthetic) image are captured under the same conditions (different conditions), and therefore belong to the same domain (different domains), and are called positive (same, negative) foreground-background pairs.

[0124] Step 4: Use the global identification results and the verification identification results to train the generator, the global discriminator, and the verification discriminator; wherein the trained generator is used to adjust the foreground or background in the initial defect image to obtain a target defect image corresponding to the first scene.

[0125] In summary, the embodiments of the present invention have at least the following features:

[0126] (A) A technique for generating defect samples even without any samples is proposed by introducing image clipping, image synthesis and harmonization techniques using an attention module, thus providing a large number of defect samples for deep learning-based defect detection.

[0127] (B) It proposes to use mixed attention units to enhance model performance, and practice shows that it can effectively enhance the model segmentation ability;

[0128] (C) Image harmonization techniques are proposed to adjust the foreground in the composite image to make it harmonious with the background.

[0129] Based on the above-mentioned embodiment, the embodiment of the present invention provides a defect zero sample generation device based on image generation method, see Fig. 9 The structure diagram of a defect zero sample generation device based on an image generation method is shown, and the device mainly includes the following parts:

[0130] An image acquisition module 902 is used to acquire a non-defective image corresponding to a first scene and a defective image corresponding to a second scene;

[0131] The mask generation module 904 is used to identify defects contained in the defect image through a pre-trained defect image matting model to generate a defect mask map;

[0132] An image synthesis module 906 is used to synthesize the defect-free image and the defect mask map to obtain an initial defect image corresponding to the first scene;

[0133] The image harmonization module 908 is used to adjust the foreground or background in the initial defect image through a pre-trained image harmonization model so that the foreground of the adjusted initial defect image is harmonious with its background, thereby obtaining a target defect image corresponding to the first scene.

[0134] The defect zero-sample generation device based on the image generation method provided by the embodiment of the present invention utilizes the defect image cutout model to generate the corresponding defect mask map based on the defect image corresponding to the second scene, and after synthesizing the defect-free image and the defect mask map, uses the image harmonization model to adjust the foreground or background of the synthesized initial defect image to make it harmonious with the background, so as to obtain a target defect image with better effect. The embodiment of the present invention utilizes image cutout, image synthesis and harmonization technology to generate defect images in the case of zero samples, thereby providing a large number of defect images with better effects for defect detection based on deep learning.

[0135] In one embodiment, the defect image matting model includes a shared encoder, a pyramid pooling unit, a semantic context branch, and an improved high-resolution detail branch; the mask generation module 904 is further used to:

[0136] Through the shared encoder, the multi-scale defect feature information corresponding to the defect image is extracted;

[0137] The first-scale defect feature information is pooled through the pyramid pooling unit, and the global context is extracted from the pooled defect feature information through the semantic context branch to obtain multi-scale context feature information and semantic-aware trimap images.

[0138] Generate a defect detail map based on the defect feature information of the second scale, the defect feature information of the third scale and the context feature information through an improved high-resolution detail branch;

[0139] The semantically-aware trimap image and the defect detail image are fused to obtain the defect mask map.

[0140] In one embodiment, the improved high-resolution detail branch includes a hybrid attention unit, a multi-level residual block and a convolutional layer; the mask generation module 904 is also used to:

[0141] Upsampling the defect feature information of the second scale and the defect feature information of the third scale respectively;

[0142] Through the hybrid attention unit, the upsampled defect feature information is extracted and fused based on the hybrid attention mechanism to obtain the fused feature information;

[0143] The fused feature information and context feature information are processed by multi-level residual blocks and convolutional layers to generate a defect detail map.

[0144] In one embodiment, the hybrid attention unit includes a channel attention subunit and a spatial attention subunit; the mask generation module 904 is further used to:

[0145] Through the channel attention subunit, feature extraction is performed on the upsampled defect feature information based on the channel attention mechanism to obtain first feature information;

[0146] Through the spatial attention subunit, feature extraction is performed on the upsampled defect feature information based on the spatial attention mechanism to obtain second feature information;

[0147] An element-wise addition operation is performed on the first feature information and the second feature information to obtain fused feature information.

[0148] In one implementation, the mask generation module 904 is further configured to:

[0149] For the residual block located at the first position, the fused feature information is input into the residual block, so that the residual block outputs defect detail features;

[0150] For other residual blocks, up-sample the context feature information so that the scale of the up-sampled context feature information is the same as that of the defect detail feature output by the previous residual block, perform feature fusion on the up-sampled context feature information and the defect detail feature output by the previous residual block, and input the fused feature into the residual block so that the residual block outputs the defect detail feature;

[0151] The defect detail features output by the last residual block and the context feature information of the same scale are fused, and the fused features are input into the convolution layer so that the convolution layer outputs a defect detail map.

[0152] In one implementation, the image synthesis module 906 is further configured to:

[0153] Performing target recognition on the defect-free image to determine the location of the target object in the defect-free image;

[0154] Based on a random scaling ratio and / or rotation angle, image processing is performed on the defect mask map to obtain a processed defect mask map;

[0155] Based on the position of the target object, the defect-free image and the processed defect mask map are synthesized to obtain an initial defect image corresponding to the first scene.

[0156] In one embodiment, the image harmonization model includes a generator, a global discriminator, and a verification discriminator; and also includes a training module for:

[0157] Generate a second defect sample based on the synthesized first defect sample and its corresponding defect mask sample by a generator;

[0158] Performing global identification on the real image corresponding to the first defect sample and the second defect sample through a global discriminator to generate a global identification result;

[0159] and, by means of a verification discriminator, performing verification and identification on the real image corresponding to the first defect sample and the second defect sample to generate a verification and identification result;

[0160] The generator, the global discriminator, and the verification discriminator are trained using the global identification results and the verification identification results; wherein the trained generator is used to adjust the foreground or background in the initial defect image to obtain a target defect image corresponding to the first scene.

[0161] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0162] An embodiment of the present invention provides an electronic device. Specifically, the electronic device includes a processor and a storage device. The storage device stores a computer program, and when the computer program is executed by the processor, it executes the method described in any one of the above-mentioned implementation methods.

[0163] Fig.10 A structural diagram of an electronic device provided in an embodiment of the present invention, the electronic device 100 includes: a processor 10, a memory 11, a bus 12 and a communication interface 13, wherein the processor 10, the communication interface 13 and the memory 11 are connected via the bus 12; the processor 10 is used to execute an executable module stored in the memory 11, such as a computer program.

[0164] The memory 11 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 13 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0165] The bus 12 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0166] Among them, the memory 11 is used to store programs, and the processor 10 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 10 or implemented by the processor 10.

[0167] The processor 10 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 10. The above processor 10 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor to execute, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 11, and the processor 10 reads the information in the memory 11 and completes the steps of the above method in combination with its hardware.

[0168] The computer program product of the readable storage medium provided in the embodiment of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be referred to the previous method embodiments, which will not be repeated here.

[0169] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0170] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for generating defect zero samples based on image generation, characterized in that: include: Acquire a non-defective image corresponding to the first scene and a defective image corresponding to the second scene; Using a pre-trained defect image cutout model, the defects contained in the defect image are identified to generate a defect mask map; synthesizing the defect-free image and the defect mask map to obtain an initial defect image corresponding to the first scene; By using a pre-trained image harmonization model, adjusting the foreground or background in the initial defect image so that the foreground of the adjusted initial defect image is in harmony with its background, thereby obtaining a target defect image corresponding to the first scene; The defect image matting model includes a shared encoder, a pyramid pooling unit, a semantic context branch, and an improved high-resolution detail branch; The step of identifying defects contained in the defect image by using a pre-trained defect image cutout model to generate a defect mask map comprises: Extracting multi-scale defect feature information corresponding to the defect image through the shared encoder; The defect feature information of the first scale is pooled by the pyramid pooling unit, and the pooled defect feature information is subjected to global context extraction by the semantic context branch to obtain multi-scale context feature information and a semantically-aware trimap. Generate a defect detail map based on the defect feature information at the second scale, the defect feature information at the third scale, and the context feature information through the improved high-resolution detail branch; Fusing the semantic perception trimap image and the defect detail image to obtain a defect mask map; The improved high-resolution detail branch includes a hybrid attention unit, a multi-level residual block and a convolutional layer; the step of generating a defect detail map based on the defect feature information of the second scale, the defect feature information of the third scale and the context feature information through the improved high-resolution detail branch includes: performing upsampling processing on the defect feature information at the second scale and the defect feature information at the third scale respectively; By means of the hybrid attention unit, feature extraction and fusion processing are performed on the upsampled defect feature information based on the hybrid attention mechanism to obtain fused feature information; The fused feature information and the context feature information are processed by multiple levels of residual blocks and convolutional layers to generate a defect detail map.

2. The defect zero sample generation method based on image generation method according to claim 1, characterized in that: The hybrid attention unit includes a channel attention subunit and a spatial attention subunit; The step of extracting and fusing the upsampled defect feature information based on the hybrid attention mechanism through the hybrid attention unit to obtain the fused feature information includes: By means of the channel attention subunit, feature extraction is performed on the upsampled defect feature information based on the channel attention mechanism to obtain first feature information; By means of the spatial attention subunit, feature extraction is performed on the upsampled defect feature information based on the spatial attention mechanism to obtain second feature information; An element-wise addition operation is performed on the first feature information and the second feature information to obtain fused feature information.

3. The defect zero sample generation method based on image generation method according to claim 1, characterized in that: The step of generating a defect detail map by processing the fused feature information and the context feature information through multiple levels of residual blocks and convolutional layers comprises: For the residual block located at the first position, inputting the fused feature information into the residual block, so that the residual block outputs defect detail features; For the other residual blocks, upsampling the context feature information so that the scale of the upsampled context feature information is the same as that of the defect detail feature output by the previous residual block, and fusion of the upsampled context feature information and the defect detail feature output by the previous residual block is performed, and the fused feature is input to the residual block so that the residual block outputs the defect detail feature; The defect detail features output by the last residual block and the context feature information of the same scale are fused, and the fused features are input into the convolution layer, so that the convolution layer outputs a defect detail map.

4. The defect zero sample generation method based on image generation method according to claim 1, characterized in that: The step of synthesizing the defect-free image and the defect mask map to obtain an initial defect image corresponding to the first scene includes: Performing target recognition on the defect-free image to determine the location of the target object in the defect-free image; Based on a random scaling ratio and / or rotation angle, performing image processing on the defect mask map to obtain a processed defect mask map; Based on the position of the target object, the defect-free image and the processed defect mask map are synthesized to obtain an initial defect image corresponding to the first scene.

5. The defect zero sample generation method based on image generation method according to claim 1, characterized in that: The image harmonization model includes a generator, a global discriminator, and a verification discriminator; The training steps of the image harmonization model include: Generate a second defect sample based on the synthesized first defect sample and its corresponding defect mask sample by the generator; Performing global identification on the real image corresponding to the first defect sample and the second defect sample by the global identifier to generate a global identification result; And, by means of the verification and identification device, the real image corresponding to the first defect sample and the second defect sample are verified and identified to generate a verification and identification result; The generator, the global discriminator and the verification discriminator are trained using the global identification result and the verification identification result; wherein the trained generator is used to adjust the foreground or background in the initial defect image to obtain a target defect image corresponding to the first scene.

6. A defect zero sample generation device based on image generation method, characterized in that: include: An image acquisition module, used to acquire a non-defective image corresponding to the first scene and a defective image corresponding to the second scene; A mask generation module, used to identify defects contained in the defect image through a pre-trained defect image matting model to generate a defect mask map; An image synthesis module, used for synthesizing the defect-free image and the defect mask map to obtain an initial defect image corresponding to the first scene; An image harmonization module, used to adjust the foreground or background in the initial defect image by using a pre-trained image harmonization model, so that the foreground of the adjusted initial defect image is harmonious with its background, and obtain a target defect image corresponding to the first scene; The defect image matting model includes a shared encoder, a pyramid pooling unit, a semantic context branch and an improved high-resolution detail branch; the mask generation module is specifically used for: Extracting multi-scale defect feature information corresponding to the defect image through the shared encoder; The defect feature information of the first scale is pooled by the pyramid pooling unit, and the pooled defect feature information is subjected to global context extraction by the semantic context branch to obtain multi-scale context feature information and a semantically-aware trimap. Generate a defect detail map based on the defect feature information at the second scale, the defect feature information at the third scale, and the context feature information through the improved high-resolution detail branch; Fusing the semantic perception trimap image and the defect detail image to obtain a defect mask map; The improved high-resolution detail branch includes a hybrid attention unit, a multi-level residual block and a convolutional layer; the mask generation module is further used for: performing upsampling processing on the defect feature information at the second scale and the defect feature information at the third scale respectively; By means of the hybrid attention unit, feature extraction and fusion processing are performed on the upsampled defect feature information based on the hybrid attention mechanism to obtain fused feature information; The fused feature information and the context feature information are processed by multiple levels of residual blocks and convolutional layers to generate a defect detail map.

7. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method according to any one of claims 1 to 5.

Images