Image generation method and apparatus, defect detection method and apparatus, and medium, device and product

Abstract

The embodiments of the present application relate to the technical field of defect detection. Disclosed are an image generation method and apparatus, a defect detection method and apparatus, and a medium, a device and a product. In the present application, threshold segmentation is first performed on an abnormal source image, the generation position of a simulated defect is obtained from a segmented image, and the segmented image is multiplied by a noise image for superposition, so as to obtain a mask image with noise data; considering a color cast distortion problem which may occur due to directly fusing a colored mask image and an original image, the original image and the noise mask image are grayed and are then fused; and as a three-channel decision diagram is obtained on the basis of the consistency between the definitions of the original image and the noise mask image and the consistency between grayscale images of the original image and the noise mask image, when the three-channel decision diagram is finally used as a fusion policy for image fusion, a colored image can be fused on the basis of preserving image definition, such that a high-quality defect sample image which restores a real state as much as possible is generated.

Description

Image generation and defect detection method, device, medium, equipment and product

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application with application number 202311754865.2 filed with the Chinese Patent Office on December 20, 2023, entitled “A method, device, medium, equipment and product for image generation and defect detection”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of defect detection technology, and specifically to an image generation and defect detection method, device, medium, equipment and product. Background Art

[0004] PCB boards will produce various defects during each process stage of production. Manufacturers in the industry widely use ADC (Automatic Defect Classification System) systems based on artificial intelligence technology to replace human labor for PCB board defect detection. The ADC system mainly uses deep learning methods to detect PCB board defects. This method requires relying on a large number of defect samples to support modeling. In actual situations, there are often not enough defect samples, which requires artificial creation of defect samples to balance the number of samples.

[0005] Defect samples need to restore their real state as much as possible, but due to the complexity of the real image itself and the diversity of detailed information, the manufactured three-channel defect sample images will have problems such as color cast and distortion, which will reduce the quality of the generated image, thereby affecting the training of the model and the effect of defect detection. Summary of the Invention

[0006] The main purpose of this application is to provide an image generation and defect detection method, device, medium, equipment and product, aiming to solve the problem of low quality of artificially created three-channel defect samples in the prior art.

[0007] To achieve the above objectives, the technical solutions adopted in the embodiments of the present application are as follows:

[0008] In a first aspect, an embodiment of the present application provides an image generation method, comprising the following steps:

[0009] Perform threshold segmentation on the abnormal source image to obtain a segmented image;

[0010] Multiply the noise image and the segmentation image based on pixel values ​​to obtain a noise mask image;

[0011] Grayscale the original image and the noise mask image to obtain a grayscale image;

[0012] According to the consistency of the clarity of the original image and the noise mask image with the grayscale image, a three-channel decision map is obtained;

[0013] According to the three-channel decision graph, the original image and the noise mask image are fused to generate the defect sample image.

[0014] In a possible implementation of the first aspect, before obtaining the three-channel decision map based on the consistency of the clarity of the original image and the noise mask image with the grayscale image, the image generation method further includes:

[0015] According to the fusion strategy of grayscale images, a grayscale decision map is obtained;

[0016] According to the consistency of the clarity of the original image and the noise mask image with the grayscale image, a three-channel decision map is obtained, including:

[0017] According to the consistency of the clarity of the original image and the noise mask image with the grayscale image, the grayscale decision map is transformed to obtain a three-channel decision map.

[0018] In a possible implementation of the first aspect, after performing threshold segmentation on the abnormal source image to obtain the segmented image, the image generation method further includes:

[0019] Obtaining a target segmentation image according to the target area image and the segmentation image;

[0020] Multiply the noise image and the segmentation image based on pixel values ​​to obtain a noise mask image, including:

[0021] The noise image and the target segmentation image are multiplied based on the pixel values ​​to obtain the noise mask image.

[0022] In a possible implementation of the first aspect, before obtaining the target segmented image based on the target area image and the segmented image, the image generation method further includes:

[0023] Binarizing the original image to obtain a first original image;

[0024] A target area image is obtained according to the first original image.

[0025] In a possible implementation of the first aspect, obtaining a target segmented image according to the target region image and the segmented image includes:

[0026] The target area image and the segmentation image are multiplied based on the pixel value to obtain the target segmentation image.

[0027] In a possible implementation of the first aspect, before performing threshold segmentation on the abnormal source image to obtain the segmented image, the image generation method further includes:

[0028] The abnormal image is sampled based on the noise generation algorithm to obtain the abnormal source image.

[0029] In a possible implementation of the first aspect, generating a defect sample image by fusing an original image and a noise mask image according to a three-channel decision graph includes:

[0030] According to the three-channel decision graph, the weighted average method is used to fuse the original image and the noise mask image to generate the defect sample image.

[0031] In a second aspect, an embodiment of the present application provides a defect detection method, comprising the following steps:

[0032] Obtaining a sample image to be detected;

[0033] The sample image to be detected is input into the defect detection model to obtain the defect detection result; wherein, the defect detection model is trained based on the original image and the defect sample image, and the defect sample image is obtained using the image generation method provided in any one of the first aspects above.

[0034] In a possible implementation of the second aspect, after inputting the target image into the defect detection model and obtaining the defect detection result, the defect detection method further includes:

[0035] According to the sample images to be detected corresponding to the misjudgment results in the defect detection results, the defect detection model is iteratively trained to obtain the target defect detection model.

[0036] In a possible implementation of the second aspect, before inputting the sample image to be detected into the defect detection model and obtaining the defect detection result, the defect detection method further includes:

[0037] According to the original image and defect sample image, the MemSeg memory-based segmentation network is trained to obtain the defect detection model.

[0038] In a possible implementation of the second aspect, the memory-based segmentation network of MemSeg includes a memory module, which is used to store memory features and compare and fuse the memory features with defect sample features.

[0039] In a third aspect, an embodiment of the present application provides an image generating device, comprising:

[0040] Segmentation module, the segmentation module is used to perform threshold segmentation on the abnormal source image to obtain a segmented image;

[0041] The superposition module is used to multiply the noise image and the segmentation image based on pixel values ​​to obtain a noise mask image;

[0042] Grayscale module, the grayscale module is used to grayscale the original image and the noise mask image to obtain a grayscale image;

[0043] The decision module is used to obtain a three-channel decision map according to the consistency of the clarity of the original image and the noise mask image with the grayscale image;

[0044] Generation module,The generation module is used to fuse the original image and the noise mask image according to the three-channel decision graph to generate the defect sample image.

[0045] In a fourth aspect, an embodiment of the present application provides a defect detection device, comprising:

[0046] An acquisition module is used to acquire a sample image to be detected;

[0047] The detection module is used to input the sample image to be detected into the defect detection model to obtain the defect detection result; wherein, the defect detection model is obtained based on the training of the original image and the defect sample image, and the defect sample image is obtained using the image generation method provided in any one of the first aspects above.

[0048] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program. When the computer program is loaded and executed by a processor, it implements the image generation method provided in any one of the first aspects above or the defect detection method provided in any one of the second aspects above.

[0049] In a sixth aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, wherein:

[0050] Memory is used to store computer programs;

[0051] The processor is used to load and execute a computer program so that the electronic device executes the image generation method provided in any one of the first aspects or the defect detection method provided in any one of the second aspects.

[0052] In the seventh aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed, is used to execute the image generation method provided in any one of the first aspects above or the defect detection method provided in any one of the second aspects above.

[0053] Compared with the prior art, the present invention has the following advantages:

[0054] The embodiments of the present application propose an image generation and defect detection method, device, medium, equipment and product. The image generation method includes: performing threshold segmentation on the abnormal source image to obtain a segmented image; multiplying the noise image and the segmented image based on pixel values ​​to obtain a noise mask image; graying the original image and the noise mask image to obtain a grayscale image; obtaining a three-channel decision diagram based on the consistency of the clarity of the original image and the noise mask image with the grayscale image; and fusing the original image and the noise mask image based on the three-channel decision diagram to generate a defect sample image. This application first performs threshold segmentation on the abnormal source image, obtains the generation position of the simulated defect from the segmented image, multiplies the segmented image with the noise image and superimposes them to obtain a mask image with noise data. Taking into account the color cast distortion problem caused by directly fusing the colored mask image with the original image, the original image and the noise mask image are first grayscaled and then fused. Since the three-channel decision graph is obtained based on the consistency of the clarity of the original image and the noise mask image with the grayscale image, when the three-channel decision graph is used as the fusion strategy for image fusion, the color image can be fused while retaining the image clarity to generate a high-quality defect sample image that restores the real state as much as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] FIG1 is a schematic diagram of the structure of an electronic device in a hardware operating environment according to an embodiment of the present application;

[0056] FIG2 is a schematic diagram of a flow chart of an image generation method provided in an embodiment of the present application;

[0057] FIG3 is a schematic diagram of an abnormal source image in the image generation method provided in an embodiment of the present application;

[0058] FIG4 is a schematic diagram of segmenting an image in the image generation method provided in an embodiment of the present application;

[0059] FIG5 is a schematic diagram of a noise image in the image generation method provided in an embodiment of the present application;

[0060] FIG6 is a schematic diagram of another noise image in the image generation method provided in an embodiment of the present application;

[0061] FIG7 is a schematic diagram of a noise mask image in the image generation method provided in an embodiment of the present application;

[0062] FIG8 is a schematic diagram of an original image in the image generation method provided in an embodiment of the present application;

[0063] FIG9 is a schematic diagram of a defect sample image in the image generation method provided in an embodiment of the present application;

[0064] FIG10 is a schematic diagram of a first original image in the image generation method provided in an embodiment of the present application;

[0065] FIG11 is a schematic diagram of a segmented image in the image generation method provided in an embodiment of the present application;

[0066] FIG12 is a schematic diagram of a flow chart of a defect detection method provided in an embodiment of the present application;

[0067] FIG13 is a schematic diagram of modules of an image generating device provided in an embodiment of the present application;

[0068] FIG14 is a schematic diagram of a module of a defect detection device provided in an embodiment of the present application;

[0069] Markings in the figure: 101 - processor, 102 - communication bus, 103 - network interface, 104 - user interface, 105 - memory. DETAILED DESCRIPTION

[0070] It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0071] The main solution of the embodiment of the present application is: threshold segmentation is performed on the abnormal source image to obtain a segmented image; the noise image and the segmented image are multiplied based on the pixel values ​​to obtain a noise mask image; the original image and the noise mask image are grayscaled to obtain a grayscale image; based on the consistency of the clarity of the original image and the noise mask image with the grayscale image, a three-channel decision diagram is obtained; based on the three-channel decision diagram, the original image and the noise mask image are fused to generate a defect sample image.

[0072] PCBs can produce a variety of defects during each manufacturing process. Factories first use automated optical inspection (AOI) equipment to capture massive amounts of images, which are then screened and filtered by human workers. However, due to the limited physical strength and capabilities of human workers, the accuracy and efficiency of image classification are low. Therefore, to save manpower and time, manufacturers in the industry are widely adopting ADC (Automatic Defect Classification) systems based on artificial intelligence technology to replace manual PCB defect detection, and this has achieved good results in actual production activities.

[0073] ADC systems primarily use deep learning to detect PCB defects. This method relies on a large number of defect samples to support modeling. However, actual production lines often do not generate sufficient defect samples, and some severe defect samples are even absent. Without defect samples, the model cannot be trained, and the ADC system cannot be implemented. Training with only a small number of defect samples inevitably results in weak feature extraction capabilities and poor recognition of unknown defects, ultimately leading to missed detections and losses for the factory.

[0074] For example, the existing ADC automatic defect classification system uses a two-stage target detection algorithm, Faster RCNN, to continuously collect large sample sets on the production line to train the defect detection model. However, the waiting period for continuous collection to reach sufficient samples is long, resulting in a large amount of redundant data, high labor consumption for manual re-evaluation and annotation, low sample classification and annotation quality, and long model training time.

[0075] In practice, defect samples are often insufficient, and some defect samples are even nonexistent. When defect samples are insufficient, the model's detection capability is very poor.

[0076] To balance the number of samples, existing technologies artificially create defective samples, often referred to as fabricating fake data. These fabricated defective samples and original, defect-free images are then used as supervised training samples to enhance the model's detection capabilities. To simulate real defective samples, the fake data must be restored to the state at the time of data acquisition, i.e., a colored image in its true state, typically an RGB three-channel image. However, in the fabrication of fake data, the images used for overlay are already colored images. Simply separating the images into three RGB channels for fusion results in color casts and distortion in the resulting image. This is especially true in PCB panel applications, where the complexity and diversity of detail in the original color images of delicate objects exacerbate the color cast and distortion, reducing the quality of the generated images and, consequently, impacting model training and application effectiveness.

[0077] To this end, the present application provides a solution. First, by performing threshold segmentation on the abnormal source image, the generation position of the simulated defect is obtained from the segmented image, and the segmented image is multiplied and superimposed with the noise image to obtain a mask image with noise data. Taking into account the color cast distortion problem caused by directly fusing the colored mask image with the original image, the original image and the noise mask image are first grayscaled and then fused. Since the three-channel decision graph is obtained based on the clarity of the original image and the noise mask image and the consistency with the grayscale image, when the three-channel decision graph is used as the fusion strategy for image fusion, the color image can be fused while retaining the image clarity to generate a high-quality defect sample image that restores the real state as much as possible.

[0078] Referring to Figure 1, Figure 1 is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiment of the present application. The electronic device may include: a processor 101, such as a central processing unit (CPU), a communication bus 102, a user interface 104, a network interface 103, and a memory 105. Among them, the communication bus 102 is used to realize the connection and communication between these components. The user interface 104 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 104 may also include a standard wired interface and a wireless interface. The network interface 103 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory 105 may optionally be a storage device independent of the aforementioned processor 101. The memory 105 may be a high-speed random access memory (RAM) memory, or a stable non-volatile memory (NVM), such as at least one disk memory. The processor 101 may be a general-purpose processor, including a central processing unit, a network processor, etc., or may be a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component.

[0079] Those skilled in the art will appreciate that the structure shown in FIG1 does not limit the electronic device and may include more or fewer components than shown, or combine certain components, or arrange components differently.

[0080] As shown in FIG1 , the memory 105 as a storage medium may include an operating system, a network communication module, a user interface module, and an image generating device.

[0081] In the electronic device shown in Figure 1, the network interface 103 is mainly used for data communication with the network server; the user interface 104 is mainly used for data interaction with the user; the processor 101 and the memory 105 in this application can be set in the electronic device, and the electronic device calls the image generation device stored in the memory 105 through the processor 101 and executes the image generation method provided in the embodiment of this application.

[0082] Referring to FIG. 2 , based on the hardware device of the aforementioned embodiment, an embodiment of the present application provides an image generation method, comprising the following steps:

[0083] S10: Perform threshold segmentation on the abnormal source image to obtain a segmented image.

[0084] In practice, the anomaly source image is an image of the simulated defect location. Regions can be manually marked on the image, or noise can be added to render the image as the anomaly source image. Threshold segmentation is then used to determine the simulated defect location. Threshold segmentation is a region-based image segmentation technique that partitions a pixel set according to grayscale. Each resulting subset forms a region corresponding to the real scene, with consistent attributes within each region, while adjacent regions do not.

[0085] Compared to manually obtaining anomaly source images, the use of noise generation algorithms can improve efficiency on the one hand, and also make the simulated defect locations more random, making the simulation effect closer to the real state. For example, before performing threshold segmentation on the anomaly source image and obtaining the segmented image, the image generation method also includes:

[0086] The abnormal image is sampled based on the noise generation algorithm to obtain the abnormal source image.

[0087] In the specific implementation process, noise generation algorithms such as Gradient Noise, Worley Noise, and more common gradient noise algorithms such as Simplex Noise and Perlin Noise are used to sample the abnormal image and generate random noise on it, thereby obtaining the abnormal source image, as shown in Figure 3. The abnormal image here does not specifically refer to an image with an abnormality, but rather to the base image for adding noise to the abnormal source image. It can be an image with or without abnormalities. The corresponding segmented image after threshold segmentation of the abnormal source image shown in Figure 3 is shown in Figure 4. Clearly, the segmented area in the image can be used as the location for adding defects.

[0088] S20: Multiply the noise image and the segmented image based on pixel values ​​to obtain a noise mask image.

[0089] In the specific implementation process, the noise image can be an image with defect noise, or it can be some texture images, or a whole-plate image formed by splicing existing defects. The purpose is to fill the position segmented from the segmented image with noise, texture, etc. that are different from the original image as a simulated defect. The filling process can be regarded as a two-image superposition process, as shown in Figures 5 and 6, which are noise images of different forms respectively. Taking the noise image shown in Figure 5 as an example, it is superimposed with the segmented image to obtain the noise mask image shown in Figure 7. The superposition adopts a pixel value multiplication method, the purpose of which is to utilize the difference between pixels in the binary segmented image. The overlapping position with the noise image in the area segmented after pixel value multiplication will be retained, thus achieving the filling of simulated defects.

[0090] S30: Grayscale the original image and the noise mask image to obtain a grayscale image.

[0091] In the specific implementation process, the original image, that is, the defect-free image taken of the PCB panel, is shown in Figure 8. Since the clarity of the image does not change with the change of its grayscale, that is, for the grayscale image and the original color image, there is only a sensory difference in color, while the detail information based on clarity is completely consistent.

[0092] S40: Obtain a three-channel decision map according to the consistency of the clarity of the original image and the noise mask image with the grayscale image.

[0093] In the specific implementation process, the three-channel decision graph is a decision graph that determines the fusion method. According to this decision graph, the image under each single layer is weighted to achieve image fusion. In order to preserve the image clarity, the clarity of the original image and the noise mask image is tested for consistency with the grayscale image. Specifically, based on the consistency of the clarity of the original image and the noise mask image with the grayscale image, before obtaining the three-channel decision graph, the image generation method also includes:

[0094] According to the fusion strategy of grayscale images, a grayscale decision map is obtained.

[0095] In the implementation, the three-channel decision graph, or the fusion strategy for color images, is associated with grayscale images to reduce the time required to directly fuse color images. Grayscaling the original image and the noise mask image can be achieved by decomposing the color image into a single-channel grayscale image using a color space conversion formula. The grayscale image's decision graph, or grayscale decision graph, is then derived using the grayscale image fusion strategy.

[0096] Based on the above steps, a three-channel decision map is obtained according to the consistency of the clarity of the original image and the noise mask image with the grayscale image, including:

[0097] According to the consistency of the clarity of the original image and the noise mask image with the grayscale image, the grayscale decision map is transformed to obtain a three-channel decision map.

[0098] In the specific implementation process, the three-channel decision map is associated with the clarity. According to the consistency of the clarity of the original image and the noise mask image with the grayscale image, fusion is performed on the basis of retaining the image clarity. The grayscale decision map under a single channel is converted to a color decision map under three channels to guide the subsequent fusion of the original image and the noise mask image.

[0099] S50: According to the three-channel decision graph, the original image and the noise mask image are fused to generate a defect sample image.

[0100] In specific implementations, image fusion schemes can be categorized as pixel-level fusion, feature-level fusion, and decision-level fusion, based on the level of information extraction, from low to high. At the decision-level, each source image has independently completed its own decision-making tasks, such as classification and recognition, before fusion. The fusion process comprehensively analyzes the results of each previous independent decision to generate a globally optimal decision and form a fused image. This approach offers advantages such as high flexibility, minimal communication traffic, optimal real-time performance, strong fault tolerance, and robust anti-interference capabilities. Figure 9 shows a defect sample image generated by fusing the original image shown in Figure 8 with the noise mask image shown in Figure 7.

[0101] More specifically, according to the three-channel decision graph, the original image and the noise mask image are fused to generate a defect sample image, including:

[0102] According to the three-channel decision graph, the weighted average method is used to fuse the original image and the noise mask image to generate the defect sample image.

[0103] In the specific implementation process, a fusion method using weighted averaging is provided. The original image and the noise mask image are given the same weights, and then the pixel values ​​of the fused image are obtained by weighted averaging. Without involving other color gamut conversions, the weighted averaging method is the simplest and most direct fusion method, and can improve the signal-to-noise ratio of the image, retain more real information on the image, and thus improve the quality of the generated defect sample image.

[0104] In this embodiment, the abnormal source image is first threshold segmented, and the generation position of the simulated defect is obtained from the segmented image. The segmented image is multiplied and superimposed with the noise image to obtain a mask image with noise data. Considering the color cast distortion problem caused by directly fusing the colored mask image with the original image, the original image and the noise mask image are first grayscaled and then fused. Since the three-channel decision graph is obtained based on the consistency of the clarity of the original image and the noise mask image with the grayscale image, when the three-channel decision graph is used as the fusion strategy for image fusion, the color image can be fused while retaining the image clarity, thereby generating a high-quality defect sample image that restores the real state as much as possible.

[0105] In one embodiment, after performing threshold segmentation on the abnormal source image to obtain a segmented image, the image generation method further includes:

[0106] Obtaining a target segmentation image according to the target area image and the segmentation image;

[0107] Multiply the noise image and the segmentation image based on pixel values ​​to obtain a noise mask image, including:

[0108] The noise image and the target segmentation image are multiplied based on the pixel values ​​to obtain the noise mask image.

[0109] In the specific implementation process, defect detection of PCB panels can be performed on the entire board image or on certain components soldered on the panel. If the latter is the case, the detection area only needs to fall within the area where the components are located, which is recorded as the target area. The image marked with the target area is the target area image. Based on the location of the target area on the target area image and the overlapping relationship of the areas segmented on the segmented image, the segmented image portion located in the target area can be selectively retained to obtain the target segmented image. Specifically, based on the target area image and the segmented image, the target segmented image is obtained, including:

[0110] The target area image and the segmentation image are multiplied based on the pixel value to obtain the target segmentation image.

[0111] In the specific implementation process, the target segmented image can also be determined by superimposing based on pixel value multiplication. The pixel value can more accurately retain the part falling within the target area. In the original image shown in Figure 8, the area covered by the components is the target area. In order to avoid interference from other colors and more clearly superimpose the target area, the original image is binarized to obtain a first original image; as shown in Figure 10, the image after the original image shown in Figure 8 is binarized, that is, the first original image. Now assume that the segmented image obtained by threshold segmentation of the abnormal source image shown in Figure 3 is shown in Figure 11, which is another example of the segmented image. According to the first original image, the target area image is obtained, and the image shown in Figure 10 is used as the target area image and multiplied with the segmented image shown in Figure 11 based on the pixel value. The area segmented on the target segmented image retains the part covered by the target area, and the segmented image shown in Figure 4 can be obtained as the target segmented image.

[0112] Referring to FIG. 12 , based on the same inventive concept as in the aforementioned embodiment, the embodiment of the present application further provides a defect detection method, comprising the following steps:

[0113] S100: Acquire a sample image to be detected;

[0114] S200: Inputting the sample image to be detected into the defect detection model to obtain the defect detection result; wherein, the defect detection model is trained based on the original image and the defect sample image, and the defect sample image is obtained using the image generation method provided in the embodiment of the present application.

[0115] During the specific implementation process, the sample images to be inspected, that is, the images of the PCB panels taken and collected on the production line, and the defect sample images generated by the image generation method provided in the embodiment of the present application are used for training the defect detection model, so that it can achieve a balance between positive and negative samples, which is conducive to supervised training to improve the effect of defect detection, and then high-quality detection can be achieved after the sample images to be inspected are input into the defect detection model.

[0116] In one embodiment, after inputting the target image into the defect detection model and obtaining the defect detection result, the defect detection method further includes:

[0117] According to the sample images to be detected corresponding to the misjudgment results in the defect detection results, the defect detection model is iteratively trained to obtain the target defect detection model.

[0118] During the specific implementation process, due to the lack of real defect samples in the training stage, the model still misjudges the detection samples after it is put into use. We can continue to collect images of samples to be detected that are misjudged on the production line, and use them to optimize the parameters of the defect detection model. After iterative training, the model's detection and testing capabilities can be further improved to obtain the target defect detection model.

[0119] In one embodiment, before inputting the sample image to be detected into the defect detection model and obtaining the defect detection result, the defect detection method further includes:

[0120] According to the original image and defect sample image, the MemSeg memory-based segmentation network is trained to obtain the defect detection model.

[0121] In the specific implementation process, a MemSeg memory-based segmentation network is provided as the network architecture of the model. Thanks to the end-to-end network structure, it also has a significant advantage in inference speed. MemSeg introduces a multi-scale feature fusion module and a spatial attention module, which can significantly improve the model accuracy of anomaly localization. The MemSeg memory-based segmentation network can also include a memory module, which is used to store memory features and compare and fuse the memory features with the features of defect samples. During the training and inference stages, by comparing the similarities and differences between the input samples and the memory features in the memory module, more effective information is provided for locating abnormal areas.

[0122] Referring to FIG. 13 , based on the same inventive concept as in the aforementioned embodiment, the embodiment of the present application further provides an image generating device, comprising:

[0123] Segmentation module, the segmentation module is used to perform threshold segmentation on the abnormal source image to obtain a segmented image;

[0124] The superposition module is used to multiply the noise image and the segmentation image based on pixel values ​​to obtain a noise mask image;

[0125] Grayscale module, the grayscale module is used to grayscale the original image and the noise mask image to obtain a grayscale image;

[0126] The decision module is used to obtain a three-channel decision map according to the consistency of the clarity of the original image and the noise mask image with the grayscale image;

[0127] Generation module,The generation module is used to fuse the original image and the noise mask image according to the three-channel decision graph to generate the defect sample image.

[0128] Referring to FIG. 14 , based on the same inventive concept as in the aforementioned embodiment, the present embodiment further provides a defect detection device, comprising:

[0129] An acquisition module is used to acquire a sample image to be detected;

[0130] The detection module is used to input the sample image to be detected into the defect detection model to obtain the defect detection result; wherein, the defect detection model is obtained based on the training of the original image and the defect sample image, and the defect sample image is obtained using the image generation method provided in any one of the first aspects above.

[0131] Those skilled in the art should understand that the division of the various modules in the embodiment is merely a division of logical functions, and in actual application, they can be fully or partially integrated into one or more actual carriers, and these modules can all be implemented in the form of software called through a processing unit, or all be implemented in the form of hardware, or in the form of a combination of software and hardware. It should be noted that the modules in the image generating device and the defect detection device in this embodiment correspond one-to-one to the steps in the image generating method and the defect detection method in the aforementioned embodiment, respectively. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned image generating method and defect detection method, and will not be repeated here.

[0132] Based on the same inventive concept as in the aforementioned embodiments, an embodiment of the present application further provides a computer-readable storage medium storing a computer program. When the computer program is loaded and executed by a processor, an image generation method or a defect detection method as provided in the embodiments of the present application is implemented.

[0133] Based on the same inventive concept as in the above embodiment, an embodiment of the present application further provides an electronic device, including a processor and a memory, wherein:

[0134] Memory is used to store computer programs;

[0135] The processor is used to load and execute a computer program so that the electronic device can execute the image generation method or defect detection method provided in the embodiments of the present application.

[0136] Based on the same inventive concept as in the aforementioned embodiments, an embodiment of the present application also provides a computer program product, including a computer program, which, when executed, is used to execute the image generation method or defect detection method provided in the embodiments of the present application.

[0137] In some embodiments, the computer-readable storage medium may be a memory device such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface mount memory, optical disk, or CD-ROM; or various devices including any one or any combination of the above memories. The computer may be various computing devices including smart terminals and servers.

[0138] In some embodiments, executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0139] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0140] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0141] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0142] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0143] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a multimedia terminal device (which can be a mobile phone, a computer, a television receiver, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0144] In summary, the present application provides an image generation and defect detection method, device, medium, equipment and product, including: performing threshold segmentation on the abnormal source image to obtain a segmented image; multiplying the noise image and the segmented image based on pixel values ​​to obtain a noise mask image; graying the original image and the noise mask image to obtain a grayscale image; obtaining a three-channel decision graph based on the clarity of the original image and the noise mask image and the consistency of the grayscale image; and fusing the original image and the noise mask image based on the three-channel decision graph to generate a defect sample image. This application first performs threshold segmentation on the abnormal source image, obtains the generation position of the simulated defect from the segmented image, multiplies the segmented image with the noise image and superimposes them to obtain a mask image with noise data. Taking into account the color cast distortion problem caused by directly fusing the colored mask image with the original image, the original image and the noise mask image are first grayscaled and then fused. Since the three-channel decision graph is obtained based on the consistency of the clarity of the original image and the noise mask image with the grayscale image, when the three-channel decision graph is used as the fusion strategy for image fusion, the color image can be fused while retaining the image clarity to generate a high-quality defect sample image that restores the real state as much as possible.

[0145] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. An image generation method, characterized in that: The following steps are involved: Perform threshold segmentation on the abnormal source image to obtain a segmented image; Multiplying the noise image and the segmented image based on pixel values ​​to obtain a noise mask image; Gray-scaling the original image and the noise mask image to obtain a grayscale image; Obtaining a three-channel decision map according to the consistency of the clarity of the original image and the noise mask image with the grayscale image; According to the three-channel decision graph, the original image and the noise mask image are fused to generate a defect sample image.

2. The image generation method according to claim 1, characterized in that: Before obtaining the three-channel decision map according to the consistency of the clarity of the original image and the noise mask image with the grayscale image, the image generation method further includes: According to the grayscale image fusion strategy, a grayscale decision map is obtained; The obtaining of a three-channel decision graph according to the consistency of the clarity of the original image and the noise mask image with the grayscale image comprises: According to the consistency of the clarity of the original image and the noise mask image with the grayscale image, the grayscale decision map is converted to obtain a three-channel decision map.

3. The image generation method according to claim 1, characterized in that: After performing threshold segmentation on the abnormal source image to obtain the segmented image, the image generation method further comprises: Obtaining a target segmented image according to the target area image and the segmented image; The step of multiplying the noise image and the segmented image based on pixel values ​​to obtain a noise mask image comprises: The noise image and the target segmented image are multiplied based on pixel values ​​to obtain a noise mask image.

4. The image generation method according to claim 3, characterized in that: Before obtaining the target segmented image according to the target area image and the segmented image, the image generation method further comprises: Binarizing the original image to obtain a first original image; The target area image is obtained according to the first original image.

5. The image generation method according to claim 3, characterized in that: The step of obtaining a target segmented image according to the target area image and the segmented image comprises: The target area image and the segmented image are multiplied based on the pixel values ​​to obtain a target segmented image.

6. The image generation method according to claim 1, characterized in that: Before performing threshold segmentation on the abnormal source image to obtain the segmented image, the image generation method further includes: The abnormal image is sampled based on a noise generation algorithm to obtain the abnormal source image.

7. The image generation method according to claim 1, characterized in that: The step of fusing the original image and the noise mask image according to the three-channel decision graph to generate a defect sample image includes: According to the three-channel decision graph, the original image and the noise mask image are fused by using a weighted average method to generate a defect sample image.

8. A defect detection method, characterized in that: The following steps are involved: Obtaining a sample image to be detected; The sample image to be detected is input into a defect detection model to obtain a defect detection result; wherein the defect detection model is obtained based on training of the original image and the defect sample image, and the defect sample image is obtained using the image generation method described in any one of claims 1-7.

9. The defect detection method according to claim 8, characterized in that: After inputting the target image into the defect detection model and obtaining the defect detection result, the defect detection method further includes: According to the sample image to be detected corresponding to the misjudgment result in the defect detection result, the defect detection model is iteratively trained to obtain a target defect detection model.

10. The defect detection method according to claim 8, characterized in that: Before inputting the sample image to be detected into the defect detection model to obtain the defect detection result, the defect detection method further includes: According to the original image and the defect sample image, the defect detection model is obtained by training with a memory-based segmentation network of MemSeg.

11. The defect detection method according to claim 10, characterized in that: The MemSeg memory-based segmentation network includes a memory module, which is used to store memory features and compare and fuse the memory features with defect sample features.

12. An image generating device, characterized in that: include: A segmentation module, wherein the segmentation module is used to perform threshold segmentation on the abnormal source image to obtain a segmented image; A superposition module, wherein the superposition module is used to multiply the noise image and the segmented image based on pixel values ​​to obtain a noise mask image; A grayscale module, wherein the grayscale module is used to grayscale the original image and the noise mask image to obtain a grayscale image; A decision module, the decision module is used to obtain a three-channel decision map according to the consistency of the clarity of the original image and the noise mask image with the grayscale image; A generation module is used to fuse the original image and the noise mask image according to the three-channel decision graph to generate a defect sample image.

13. A defect detection device, characterized in that: include: An acquisition module, the acquisition module is used to acquire a sample image to be detected; A detection module, wherein the detection module is used to input the sample image to be detected into a defect detection model to obtain a defect detection result; wherein the defect detection model is obtained based on training of the original image and the defect sample image, and the defect sample image is obtained using the image generation method described in any one of claims 1-7.

14. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is loaded and executed by a processor, the image generation method according to any one of claims 1 to 7 or the defect detection method according to any one of claims 8 to 11 is implemented.

15. An electronic device, characterized in that: comprising a processor and a memory, wherein: The memory is used to store computer programs; The processor is used to load and execute the computer program so that the electronic device executes the image generation method according to any one of claims 1 to 7 or the defect detection method according to any one of claims 8 to 11.

16. A computer program product, characterized in that It comprises a computer program, which, when executed, is used to execute the image generation method according to any one of claims 1 to 7 or the defect detection method according to any one of claims 8 to 11.

Images