Image inpainting and defect detection method and system based on generative adversarial network

Abstract

The present application relates to the technical field of industrial machine vision detection and data processing, in particular to an image repairing and defect detection method and system based on a generative adversarial network, which repairs and detects a to-be-detected image through a detection and repair model, including a double-branch feature extraction network and a generative adversarial network, to obtain a repaired image after removing reflection interference and a defect recognition result; the double-branch feature extraction network extracts reflection area and defect semantic prior information from the to-be-detected image to obtain a reflection probability mask graph and a defect prior information set; the generative adversarial network takes the defect prior information set as a guide constraint, repairs and detects the to-be-detected image according to the marked reflection area in the reflection probability mask graph, and obtains the repaired image after removing the reflection interference and the defect recognition result. The present application can accurately repair the reflection and retain the defects, and avoid over-smoothing or false modification of the defect area when removing the reflection.

Description

Technical Field

[0001] This invention relates to the field of industrial machine vision inspection and data processing technology, specifically to an image inpainting and defect detection method and system based on generative adversarial networks. Background Technology

[0002] In industrial product quality inspection, uneven reflections are easily generated on the glass surfaces of photovoltaic panels, the protective films of OLED screens, and the smooth surfaces of metal parts. Traditional machine vision inspection algorithms face two main problems when dealing with these reflective areas: first, the bright spots formed by reflections can mask the actual defects (such as scratches, bubbles, and dents) located beneath them, leading to missed detections; second, the complex light spots formed by reflections are similar in shape to certain defects, making them easy to misidentify as defects, resulting in over-detection.

[0003] Traditional image processing techniques, such as histogram equalization, homomorphic filtering, or multi-angle lighting, can alleviate the effects of reflections to some extent, but they cannot fundamentally restore the image details covered by reflections. In particular, they often perform poorly in high-speed inspection scenarios on automated production lines.

[0004] Therefore, there is an urgent need for an image inpainting and defect detection method and system based on generative adversarial networks, which can perform accurate repair of defects by eliminating reflections and preserving defects, avoiding excessive smoothing or incorrect modification of defect areas when removing reflections, so as to solve the problem of missed detection caused by reflections masking real defects. Summary of the Invention

[0005] One of the objectives of this invention is to provide an image inpainting and defect detection method based on generative adversarial networks, which can perform accurate repair of defects by eliminating reflections and preserving defects, avoiding excessive smoothing or incorrect modification of defect areas when removing reflections, so as to solve the problem of missed detection caused by reflections masking real defects.

[0006] The basic solution provided by this invention is an image inpainting and defect detection method based on generative adversarial networks, comprising: By constructing a detection and repair model, the image to be detected is repaired and inspected, and the repaired image after removing reflection interference and the defect identification results are obtained. The detection and repair model includes: a two-branch feature extraction network and a generative adversarial network; A dual-branch feature extraction network is used to extract reflective regions and defect semantic prior information from the image to be detected, and to obtain a reflective probability mask and a defect prior information set. The reflective probability mask is used to mark reflective regions. The defect prior information set includes: a defect prior heatmap, used to mark suspected defect regions, and a defect semantic feature vector, used to provide morphological prior information of suspected defects. Generative adversarial networks (GANs) are used to repair and detect images based on a set of prior information about defects, using the marked reflective areas in a probability mask image to obtain the repaired image after removing reflective interference and the defect recognition results.

[0007] The second objective of this invention is to provide an image inpainting and defect detection system based on generative adversarial networks, which can perform precise repair of defects by eliminating reflections and preserving defects, avoiding excessive smoothing or incorrect modification of defect areas when removing reflections, thereby solving the problem of missed detection caused by reflections masking real defects.

[0008] This invention provides a second basic solution: an image inpainting and defect detection system based on generative adversarial networks, comprising: computer equipment; The computer equipment uses a constructed detection and repair model to repair and detect the image to be inspected, and obtains the repaired image after removing reflection interference and the defect identification results. The detection and repair model includes: a two-branch feature extraction network and a generative adversarial network; A dual-branch feature extraction network is used to extract reflective regions and defect semantic prior information from the image to be detected, and to obtain a reflective probability mask and a defect prior information set. The reflective probability mask is used to mark reflective regions. The defect prior information set includes: a defect prior heatmap, used to mark suspected defect regions, and a defect semantic feature vector, used to provide morphological prior information of suspected defects. Generative adversarial networks (GANs) are used to repair and detect images based on a set of prior information about defects, using the marked reflective areas in a probability mask image to obtain the repaired image after removing reflective interference and the defect recognition results.

[0009] Beneficial effects: This scheme constructs a detection and repair model to repair and detect images to be inspected, and obtains repaired images after removing reflection interference and defect identification results; the detection and repair model consists of a dual-branch feature extraction network and a generative adversarial network; Specifically, the dual-branch feature extraction network decouples reflective region segmentation and defect prior extraction into two collaborative tasks and introduces a cross-branch feature interaction mechanism. The reflective probability mask generated by the first branch is used to mark reflective regions, accurately locating reflective interference regions that need repair, providing a clear operational range constraint for the generative adversarial network. The defect prior heatmap and defect semantic feature vector extracted by the second branch under reflective interference are used to mark suspected defective regions and provide morphological prior information of suspected defects, respectively, thus providing the generative adversarial network with spatial hints about where defects might exist and morphological guidance on what defects should look like. Furthermore, the design of the dual-branch feature extraction network makes the two tasks of reflective segmentation and defect prior extraction mutually reinforcing and complementary, rather than simply parallel.

[0010] In the process of repairing and detecting images, generative adversarial networks repair and detect based on the marked reflective areas in the reflective probability mask. At the same time, they are protected by spatial attention and guided by the defect prior heatmap in the defect prior information set and the defect semantic feature vector. This effectively avoids the excessive smoothing or erroneous modification of defect areas when removing reflections in traditional repair methods. It achieves the accurate repair effect of repairing reflections without repairing defects, and fundamentally solves the problem of missed detection caused by reflections masking real defects.

[0011] In summary, this solution can perform precise repair of defects that retain reflectivity, avoiding excessive smoothing or incorrect modification of the defect area when removing reflectivity, thus solving the problem of missed detection caused by reflectivity masking the true defects. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating an embodiment of the image inpainting and defect detection method based on generative adversarial networks of the present invention. Detailed Implementation

[0013] The following detailed description illustrates the specific implementation method: Example 1 This embodiment provides an image inpainting and defect detection method based on generative adversarial networks, as shown in the attached figure. Figure 1 As shown, it includes the following: The constructed detection and repair model is used to repair and detect the image to be detected, and to obtain the repaired image after removing reflection interference and the defect identification results; the defect identification results include: the classification and location information of the defects; The detection and repair model includes: a two-branch feature extraction network and a generative adversarial network; A dual-branch feature extraction network is used to extract reflective regions and defect semantic prior information from the image to be detected, and to obtain a reflective probability mask map and a defect prior information set; wherein the defect prior information set includes: a defect prior heatmap and a defect semantic feature vector. The dual-branch feature extraction network includes: a shared coding backbone network, a reflective region segmentation branch, and a defect prior extraction branch; A shared coding backbone network is used to perform multi-layer convolutions on the input image to be detected, generating multi-scale feature maps, including: low-level texture features. (High resolution), mid-level semantic features and high-level semantic features (Lower resolution, but the field of view is larger). A deep convolutional neural network is used as the shared coding backbone network; in this embodiment, a residual network ResNet-50 is preferably used, and its final global average pooling and fully connected layers are removed. The input is the image to be detected, i.e., the original RGB image affected by reflections. The dimensions are 256×256×3; The processing procedure is as follows: Image The feature maps are sequentially processed through five convolutional stages (Conv1 to Conv5) of ResNet-50 to generate feature maps of different scales. The outputs of the last three stages (Conv3, Conv4, and Conv5) are denoted as... These correspond to low-level texture features, mid-level semantic features, and high-level semantic features, respectively.

[0014] The reflective region segmentation branch, as the first branch, is used to separate the reflective region from the background based on the multi-scale feature map and obtain the reflective probability mask map. The first branch uses a semantic segmentation network (such as U-Net) structure to identify and segment reflective regions in the input image, and generates a reflective probability mask map of the same size as the input image (the image to be detected), which is used to mark which regions are affected by reflective interference, i.e., to mark reflective regions; Specifically, the first branch performs feature fusion on the multi-scale feature map to generate multi-scale pyramid features; Multi-scale pyramid features are upsampled and fused to obtain a feature map that incorporates global contextual information; Based on the feature map that incorporates global contextual information, a spatial attention weight map is generated through the spatial attention module. The spatial attention weight map and the feature map that incorporates global contextual information are multiplied element-wise to obtain the enhanced reflective features. The reflective features are reduced in dimension and then upsampled to restore the original input size. After activation by an activation function, a reflective probability mask is obtained.

[0015] The first branch is responsible for accurately separating the reflective area from the background. Its semantic segmentation network structure combines a Feature Pyramid Network (FPN) with a spatial attention module. In this embodiment, it is specifically as follows: Multi-scale feature maps output by the shared coding backbone network The input is the FPN, which passes high-level semantic information to the lower levels through a top-down path and lateral connections, generating multi-scale pyramid features. ; pyramid features After upsampling and Fusion, then upsampling and P3 fusion ultimately yields a feature map that incorporates global contextual information. ; To more accurately locate the boundaries of highlight areas, a spatial attention module is introduced. Two two-dimensional spatial descriptors are generated by max pooling and average pooling, respectively. These descriptors are then concatenated, and the concatenated descriptor is passed through a 7×7 convolutional layer with sigmoid activation to generate a spatial attention weight map. ; and Element-wise multiplication yields the enhanced reflective features. ; Will The input is reduced in dimensionality using a 1×1 convolutional layer, then upsampled using bilinear interpolation to restore it to the original input size, and finally activated by the Sigmoid function to output a reflective probability mask. Used to indicate the location and intensity of reflective interference; Each pixel value in the graph is between 0 and 1, indicating the probability that the pixel belongs to a reflective area. The closer the value is to 1, the stronger the reflective interference.

[0016] The defect prior extraction branch, as the second branch, is used to generate a spatial attention weight map during the process of separating the reflective region from the background based on the multi-scale feature map and the reflective region segmentation branch. This captures defect cues, extracts defect semantic prior information, and generates a defect prior heatmap and defect semantic feature vector as a defect prior information set.

[0017] The second branch, the object detection or feature extraction network (such as FPN or RPN), is responsible for initially capturing potentially obscured defect clues, i.e., possible defect features, in the original image with strong reflective interference. Due to the presence of reflection, this branch does not pursue precise defect classification, but focuses on extracting a heatmap of the probability of defect existence and boundary clues of defects. Its output defect prior information set includes the location and morphological prior information of suspected defects, where the morphological prior information will serve as a constraint condition for subsequent repair. Its structure adopts a Region Proposal Network (RPN), which is specifically described in this embodiment as follows: Perform cross-branch feature interaction, and use the spatial attention weight map generated by the first branch. (or its downsampled version) and mid-level semantic features of the shared coding backbone network Element-wise multiplication is performed to obtain the feature map after reflection enhancement; the purpose of this operation is to force the model to reflect in the reflective region (i.e., Pay more attention to searching for potential defect features in areas with high values; The feature map enhanced by reflection is input into a lightweight RPN. The RPN uses a sliding window to pre-set anchor boxes of different scales and aspect ratios at each location, and outputs the confidence and offset of the suspected defect region for each anchor box. The anchor boxes are candidate boxes with fixed scales and aspect ratios pre-generated at each location of the feature map in the RPN. The confidence level of the suspected defect area, i.e. the foreground / background confidence level, determines whether the anchor point box contains an object suspected of being defective. Suspected defects include actual defects as well as reflective spots that may be misjudged as defects. Offset is used for coarse bounding box regression, which fine-tunes the position of the anchor box to better fit the outline of the suspected defect area; the adjustment amount is the offset value, and in RPN, each anchor box outputs a four-dimensional offset vector. The confidence scores of all suspected defect regions output by the RPN are mapped back to the image space, where the image space refers to the pixel coordinate space of the original input image (the image to be detected). This can be understood as mapping the confidence scores output by the RPN from their positions on the feature map back to their corresponding positions on the original input image (e.g., 256×256 pixels). Specifically, for each pixel position in the image, the maximum confidence score of all suspected defect proposal boxes passing through that position is counted, generating a defect prior heatmap. It is used to provide spatial distribution information of suspected defect areas; the proposal box is the finely tuned candidate box output by RPN after offset regression and confidence filtering of anchor boxes; Simultaneously, the convolutional features within these proposal boxes are extracted and global max pooling is performed to obtain a fixed-dimensional defect semantic feature vector. It is used to provide morphological prior information about defects (what the defects should look like).

[0018] Output: Defect prior heatmap With defect semantic feature vector Together, they serve as the output of the defect prior extraction branch, denoted as the defect prior information set. .in, This will serve as a constraint on spatial location in subsequent steps. This will serve as a semantic prior when the subsequent generator recovers the defect details.

[0019] In summary, the output of the dual-branch feature extraction network includes: Reflection probability mask From the first branch, used to indicate where repairs are needed, marking reflective areas; Defect prior information set From the second branch, including: defect prior heatmap Used to indicate where defects may exist, to indicate suspected defect areas, and as a defect semantic feature vector. It is used to indicate what a defect should look like and to provide morphological prior information about suspected defects.

[0020] Generative adversarial networks are used to repair and detect images based on a probability mask of reflective material and a set of prior information about defects, obtaining repaired images after removing reflective interference and defect recognition results.

[0021] Specifically, generative adversarial networks include: a generator and a discriminator; The generator employs an encoding / decoding structure with an attention mechanism. Its decoding process is guided by a defect prior information set, performing pixel reconstruction only within the reflective regions corresponding to the reflective probability mask. For non-reflective regions and regions marked as potentially defective by the defect prior information set (defect prior heatmap), a larger penalty is applied. This achieves the goal of removing reflections while preserving or restoring the true defect features to the maximum extent possible. The larger penalty is applied by assigning higher weight coefficients to non-reflective and potentially defective regions in the total loss function. For example, for non-reflective regions, the generator output must be completely identical to the pixel values ​​of the original image; otherwise, a weight greater than a preset threshold is applied. For suspected defective areas, a high weight is also applied to the loss, forcing the generator not to modify the pixels in that area.

[0022] Specifically, U-Net is used as the backbone network, and attention gates are introduced at its skip connections to suppress irrelevant background features and focus on the reflective restoration area; the encoding and decoding structure includes an encoder and a decoder. The generator's input includes: the image to be detected. Reflection probability mask Defect Prior Heat Map Defect semantic feature vector ; The specific processing procedure is as follows: Input layer, used to generate a reflectivity probability mask. The image is stitched together with the image to be detected along the channel dimension to generate a stitched feature map, which guides the network to focus on the areas that need to be repaired; Defect Prior Heatmap in Encoder In the deeper stages of the encoder, the feature map is fused with it element-wise through multiplication, serving as a spatial attention weight to prompt the generator to adjust its approach when fixing reflections. High-response regions (i.e., areas where defects may exist) are specially protected to avoid over-smoothing. In U-Net encoders, there are usually multiple downsampling layers. The deep stage refers to the high semantic level near the bottleneck layer, such as the fourth downsampling layer of U-Net (the layer with the lowest resolution and the most channels). In contrast, the shallow stage refers to the first and second downsampling layers near the input.

[0023] Defect semantic feature vector in decoder Instead of simple concatenation, it injects data into each level of the decoder through an Adaptive Instance Normalization (AdaIN) layer. Specifically, After transformation by a fully connected layer, affine parameters are generated to adjust the scale and shift of the decoder feature map, thereby guiding the generator at the semantic level to restore the true texture and contour of the defects.

[0024] The discriminator employs a PatchGAN structure; its unique feature is that, in addition to judging the authenticity of input image patches, it introduces an auxiliary classifier. This auxiliary classifier generates suspected defect regions in the image (based on a prior defect heatmap). The high-response region (guided localization) is extracted and compared with a pre-established standard defect feature library to output a morphological similarity score; in this embodiment, the standard defect feature library is the photovoltaic glass standard defect feature library. In PatchGAN, an image patch refers to dividing the entire image into multiple N×N small blocks (e.g., 70×70), and the discriminator independently judges the authenticity of each small block; the input image patch refers to the divided small block; while in the auxiliary classifier of this scheme, the image patch specifically refers to the suspected defect region cropped from the repaired image.

[0025] Specifically, the discriminator adopts a dual-task discriminant structure: The first task is to determine whether the repaired image output by the generator meets the preset image distribution requirements, that is, a real and clear image distribution (real / fake discrimination); specifically, in this embodiment, the first task outputs a scalar (0~1) to represent the probability that the input image is a real image or a generated image; The second task involves cropping or extracting the reflective regions indicated by the reflectivity probability mask in the repaired image, comparing them with image patches in a pre-stored standard defect sample library, and calculating the similarity between the repaired defect region and the standard defect samples in features such as contour, grayscale distribution, and texture. This ensures that the repair process does not distort the original shape of the defect. The similarity score serves as the input to the loss function of the discriminator's second task. During the training phase, this similarity score is used to calculate... The loss guides the generator to learn to produce morphologically realistic defects; during the inference phase, this similarity can be used as part of the output to represent the credibility of the repaired defect; specifically, in this embodiment, the second task outputs a morphological similarity score (0~1), as well as optional defect category labels and location coordinates.

[0026] A dataset was constructed to train the detection and repair model. In this embodiment, 5,000 images from the photovoltaic glass production line were collected, including: images with defects such as scratches, dirt, and chipping, and irregular reflections on the surface; and corresponding standard images without defects and without reflections. Using annotation tools, reflective areas in all images are annotated at the pixel level to generate reflective area label maps; at the same time, the location and type of defects in the images are annotated to generate defect labels, and a dataset is constructed, including a training set and a test set.

[0027] The dual-branch network and the generative adversarial network are jointly trained using the training set. During model training, the Adam optimizer is used for end-to-end training. The total loss function is defined as follows: in, To combat the loss, ensure that the generator deceives the discriminator and makes the output image realistic; To the difference between the generated image and the real reflectionless image Distance loss (ensuring background fidelity) ensures that the restored image is consistent with the real non-reflective image (if paired data is available) or image background at the pixel level; The defect feature matching loss (ensuring the repaired defect matches the standard library) forces the generator's output defect region features to match the feature distribution in the standard defect sample library; specifically, it calculates the feature distribution in the generated image, which is composed of... The features extracted from the indicated high-response region are compared with the feature vectors in the standard defect sample library. Cosine similarity between them; through experiments, Set to 100. Set it to 50.

[0028] In practical applications, the trained model is deployed to the industrial control computer on the production line. When the image acquisition device acquires the image of the photovoltaic panel to be tested, it directly inputs the detection and repair model and outputs the repaired non-reflective image, i.e. the repaired image. At the same time, the defect identification results are output, including the defect classification and location information.

[0029] First, this scheme decouples reflective region segmentation and defect prior extraction into two collaborative tasks by constructing a dual-branch feature extraction network, and introduces a cross-branch feature interaction mechanism. The reflective mask map generated by the first branch accurately locates the interference region that needs to be repaired, providing a clear operational range constraint for the generator; the defect prior heatmap extracted by the second branch under reflective interference... and defect semantic feature vector This provides the generator with spatial cues indicating "where defects might exist" and morphological guidance on "what defects should look like." During decoding, the generator only reconstructs pixels within the area indicated by the reflective mask, and is simultaneously influenced by... Spatial attention protection and The semantic feature guidance effectively avoids the excessive smoothing or incorrect modification of defect areas when removing reflections by traditional repair methods, and achieves the precise repair effect of repairing reflections without repairing defects, fundamentally solving the problem of missed detection caused by reflections masking real defects.

[0030] Secondly, this scheme innovatively designs a dual-task discrimination structure in the discriminator, which not only retains the traditional GAN's ability to discriminate the overall realism of the image, but also introduces a defect morphology discrimination task based on a standard defect sample library. This discriminator will identify suspected defect areas (by...) in the repaired image. The generator compares the repaired image with a pre-established standard defect feature library, calculating morphological similarity from multiple dimensions such as contour, grayscale distribution, and texture. This design forces the generator to ensure that the generated defect area closely approximates the distribution of real defect samples in the feature space when repairing reflections, rather than simply generating a visually natural image. The resulting repaired image has morphological features of defect areas that are highly consistent with real defects, allowing it to be directly input into a subsequent defect classifier for accurate identification and measurement, avoiding secondary misjudgments caused by distortion of defect morphology during the repair process.

[0031] Finally, this scheme integrates the dual-branch feature extraction network and the generative adversarial network into a unified end-to-end joint training framework, and optimizes them using a composite loss function consisting of adversarial loss, content-aware loss, and defect prior loss. During training, the dual-branch networks not only learn their respective tasks but also achieve information complementarity through feature interaction—the attention weights of the reflectivity segmentation branch guide the defect prior branch to focus its search on highly reflective areas, while the features extracted by the defect prior branch, in turn, constrain the generator's repair behavior. This collaborative optimization mechanism enables the model to adaptively handle complex reflectivity interference of different types and intensities, as well as various defects of diverse shapes. It exhibits excellent generalization performance and robustness in various high-reflectivity surface detection tasks such as photovoltaic glass, OLED screens, and metal parts, without requiring cumbersome parameter adjustments for specific production lines.

[0032] Compared with existing technologies, this solution differs from the traditional single repair or detection architecture by constructing a dual-branch collaborative feature extraction mechanism. Existing technologies typically employ end-to-end generative networks for direct reflection removal or use segmentation networks alone to locate reflective areas, lacking defect information to guide the repair process. This solution constructs a dual-branch structure with a shared backbone network and introduces cross-branch feature interaction—multiplying the spatial attention weights generated by the first branch with the defect extraction process of the second branch, forcing the model to invest more attention in searching for potential defects in reflective areas. This makes the two tasks of reflection segmentation and defect prior extraction mutually reinforcing and complementary, rather than simply parallel. This solution differs from conventional image inpainting generators by implementing multimodal guided inpainting based on prior defects. Existing GAN-based inpainting methods typically only use the original image and mask as input. The generator lacks semantic understanding of the original content within the inpainting area, easily removing or smoothing defects along with reflections. This proposed method innovatively accepts four parts of input and adopts a differentiated fusion strategy for different prior information. As spatial attention weights, they are multiplied element-wise. The decoder is injected through the AdaIN layer; this multimodal guidance mechanism of spatial guidance and semantic injection enables the generator to accurately identify and protect suspected defect areas when repairing reflections.

[0033] This scheme differs from traditional single-task image discriminators by constructing a morphological discrimination mechanism based on a standard defect library; Existing GAN discriminators typically only judge the overall realism of the generated image, failing to constrain the morphological fidelity of the repaired defective region. This proposed discriminator employs a dual-task structure, introducing an auxiliary classifier in addition to the conventional realism / falsity discrimination to classify the morphological fidelity of the repaired image from the defective region. The suspected defect areas are extracted and compared with the pre-built standard defect feature library. The morphological similarity of contours, textures, etc. is calculated. The authenticity of the defect morphology is taken as an independent optimization target to ensure that the repaired defects not only look natural, but also conform to the distribution of real defects in the feature space.

[0034] Example 2 This embodiment provides an image inpainting and defect detection system based on generative adversarial networks (GANs) for performing the aforementioned image inpainting and defect detection method based on GANs, and includes: a computer device; The computer equipment uses a constructed detection and repair model to repair and detect the image to be inspected, and obtains the repaired image after removing reflection interference and the defect identification results. The computer equipment includes, but is not limited to, industrial control computers and servers; in this embodiment, an industrial control computer is used, and the image to be detected is acquired by an image acquisition device (such as a camera).

[0035] The trained detection and repair model is deployed to the industrial control computer on the production line. When the camera captures the image of the photovoltaic panel to be tested, it is directly input into the detection and repair model. The detection and repair model outputs the repaired image after removing reflection interference and the defect recognition result.

[0036] The above descriptions are merely embodiments of the present invention. Commonly known structures and characteristics are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are aware of all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. An image inpainting and defect detection method based on generative adversarial networks, characterized in that, include: By constructing a detection and repair model, the image to be detected is repaired and inspected, and the repaired image after removing reflection interference and the defect identification results are obtained. The detection and repair model includes: a two-branch feature extraction network and a generative adversarial network; A dual-branch feature extraction network is used to extract reflective regions and defect semantic prior information from the image to be detected, and to obtain a reflective probability mask and a defect prior information set. The reflective probability mask is used to mark reflective regions. The defect prior information set includes: a defect prior heatmap, used to mark suspected defect regions, and a defect semantic feature vector, used to provide morphological prior information of suspected defects. Generative adversarial networks (GANs) are used to repair and detect images based on a set of prior information about defects, using the marked reflective areas in a probability mask image to obtain the repaired image after removing reflective interference and the defect recognition results.

2. The image inpainting and defect detection method based on generative adversarial networks according to claim 1, characterized in that, The dual-branch feature extraction network includes: a shared coding backbone network, a reflective region segmentation branch, and a defect prior extraction branch; A shared coding backbone network is used to perform multi-layer convolution on the input image to be detected, generating multi-scale feature maps, including: low-level texture features, mid-level semantic features and high-level semantic features; The reflective region segmentation branch is used to separate the reflective region from the background based on the multi-scale feature map and obtain the reflective probability mask map; The defect prior extraction branch is used to extract the defect semantic prior information based on the spatial attention weight map generated during the process of separating the reflective region from the background according to the reflective region segmentation branch, and to generate the defect prior heatmap and defect semantic feature vector as the defect prior information set.

3. The image inpainting and defect detection method based on generative adversarial networks according to claim 2, characterized in that, The step of separating the reflective region from the background based on the multi-scale feature map to obtain a reflective probability mask map includes: For multi-scale feature maps, feature fusion is performed to generate multi-scale pyramid features; Multi-scale pyramid features are upsampled and fused to obtain a feature map that incorporates global contextual information; Based on the feature map that incorporates global contextual information, a spatial attention weight map is generated through the spatial attention module. The spatial attention weight map and the feature map that incorporates global contextual information are multiplied element-wise to obtain the enhanced reflective features. The reflective features are reduced in dimension and then upsampled to restore the original input size. After activation by an activation function, a reflective probability mask is obtained.

4. The image inpainting and defect detection method based on generative adversarial networks according to claim 3, characterized in that, The spatial attention weight map generated during the process of separating the reflective region from the background based on the reflective region segmentation branch is used to extract defect semantic prior information, generate a defect prior heatmap and defect semantic feature vector, and serve as a defect prior information set, including: Cross-branch feature interaction is performed by multiplying the spatial attention weight map generated by the reflective region segmentation branch with the mid-layer semantic features of the shared coding backbone network element-wise to obtain the feature map after reflective enhancement. The feature map after reflection enhancement is input into the region proposal network. The region proposal network uses a sliding window to pre-set anchor boxes of different scales and aspect ratios at each location, and outputs the confidence and offset of the suspected defect region for each anchor box. Map the confidence scores of all suspected defective regions output by the region proposal network back to the image space. For each pixel location in the image, calculate the maximum confidence score of all suspected defective proposal boxes passing through that location to generate a defective prior heatmap. Simultaneously, the convolutional features within the proposal box are extracted and global max pooling is performed to obtain a fixed-dimensional defect semantic feature vector.

5. The image inpainting and defect detection method based on generative adversarial networks according to claim 1, characterized in that, Generative adversarial networks consist of a generator and a discriminator. The generator employs an encoding / decoding structure with an attention mechanism. Its decoding process is guided by a set of prior defect information. Pixel reconstruction is performed within the reflective region corresponding to the reflective probability mask. Modification penalties are applied to non-reflective regions and suspected defective regions. The discriminator, using the PatchGAN structure, judges the authenticity of input image patches and sets up an auxiliary classifier to extract suspected defect regions in the generated image, compares them with a pre-established standard defect feature library, and outputs a morphological similarity score.

6. The image inpainting and defect detection method based on generative adversarial networks according to claim 5, characterized in that, The discriminator employs a dual-task discriminator structure: First task: Determine whether the repaired image output by the generator meets the preset image distribution requirements; The second task is to crop or extract the reflective areas indicated by the reflective probability mask in the repaired image, compare them with the image blocks in the pre-stored standard defect sample library, and calculate the similarity between the repaired defect area and the standard defect sample in terms of preset features.

7. The image inpainting and defect detection method based on generative adversarial networks according to claim 1, characterized in that, Also includes: Build a dataset and train the detection and repair model; The dataset is used to jointly train the dual-branch network and the generative adversarial network.

8. The image inpainting and defect detection method based on generative adversarial networks according to claim 7, characterized in that, During the training process, the Adam optimizer is used for end-to-end training, and the total loss function is: in, To combat the losses; To the difference between the generated image and the real reflectionless image Distance loss; For defect feature matching loss; and These are the weighting coefficients.

9. An image inpainting and defect detection system based on generative adversarial networks, characterized in that, include: Computer equipment; The computer equipment uses a constructed detection and repair model to repair and detect the image to be inspected, and obtains the repaired image after removing reflection interference and the defect identification results. The detection and repair model includes: a two-branch feature extraction network and a generative adversarial network; A dual-branch feature extraction network is used to extract reflective regions and defect semantic prior information from the image to be detected, and to obtain a reflective probability mask and a defect prior information set. The reflective probability mask is used to mark reflective regions. The defect prior information set includes: a defect prior heatmap, used to mark suspected defect regions, and a defect semantic feature vector, used to provide morphological prior information of suspected defects. Generative adversarial networks (GANs) are used to repair and detect images based on a set of prior information about defects, using the marked reflective areas in a probability mask image to obtain the repaired image after removing reflective interference and the defect recognition results.

10. The image inpainting and defect detection system based on generative adversarial networks according to claim 9, characterized in that, The computer equipment includes: industrial control computers and servers.

Images