Neural network training method and device, defect detection method and device and storage medium

By reconstructing the network and feature channel screening modules, the training defect image is generated and reconstructed, the problem of accuracy and robustness of defect detection of different types of products or objects in the prior art is solved, and the data collection and labeling process is reduced , improves detection efficiency and user experience.

CN119027361APending Publication Date: 2024-11-26RICOH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310587811.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-23
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Existing neural network defect detection methods are difficult to achieve accurate and robust detection of disordered defects in different types of products or objects, and require a large amount of labeled data and manual labeling process.

Method used

By introducing the reconstruction network and feature channel screening module, training defect images are generated and reconstructed, feature channels and spatial relationships are extracted, and neural network parameters are adjusted to achieve defect detection.

Benefits of technology

It realizes accurate generation and robust detection of disordered defects in different types of products or objects, reducing the collection and manual labeling of defect data, and improving user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027361A_ABST
    Figure CN119027361A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a neural network training method, a defect detection method, a defect detection device and a medium. The neural network training method comprises the following steps: inputting a training image, and calculating a perceptible graph of the training image; generating a training defect image from at least the training image and the perceptible map of the training image, the training defect image comprising one or more defects generated based on the perceptible map of the training image; reconstructing the training defect image through a reconstruction network comprising a first sub-module to obtain a reconstructed training reconstruction image without defects and a perceptible graph of the reconstructed training image, wherein the first sub-module is used for screening and processing features by screening feature channels and extracting spatial relationships of the channels; and performing defect detection according to the training image, the training reconstruction image and the perceptible graph of the training reconstruction image so as to train the neural network and adjust parameters of the neural network.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a neural network training method, and a method, device and computer-readable storage medium for defect detection using a neural network. Background Art

[0002] Defect detection is an important factor in product quality control. Since various defects will continue to occur during production, it is usually impossible to enumerate the various situations in which defects occur. How to robustly and automatically detect these defects is an urgent problem to be solved.

[0003] In current defect detection methods, a large amount of labeled data is usually used to train the neural network for defect detection, and training and defect detection are performed separately for each type of product or object. However, this neural network training method and defect detection method using this neural network cannot achieve accurate generation and robust detection of disordered defects of different types of products or objects.

[0004] Therefore, there is a need for an improved neural network training method and a defect detection method for accurately detecting image defects. Summary of the invention

[0005] To solve the above technical problems, according to one aspect of the present invention, a neural network training method is provided, comprising: inputting a training image and calculating a perceptible map of the training image; generating a training defect image at least based on the training image and the perceptible map of the training image, wherein the training defect image comprises one or more defects generated based on the perceptible map of the training image; reconstructing the training defect image through a reconstruction network including a first submodule to obtain a reconstructed training reconstructed image with defects removed and a perceptible map of the training reconstructed image, wherein the first submodule is used to screen and process features by screening feature channels and extracting spatial relationships between each channel; and performing defect detection based on the training image, the training reconstructed image and the perceptible map of the training reconstructed image to train the neural network and adjust the parameters of the neural network.

[0006] According to another aspect of the present invention, there is provided a defect detection method, comprising: inputting an image to be detected, reconstructing the image to be detected by a reconstruction network including a first submodule, obtaining a reconstructed image with defects removed after reconstruction and a perceptible image of the reconstructed image, wherein the first submodule is used to screen and process features by screening feature channels and extracting spatial relationships of each channel; and performing defect segmentation according to the image to be detected, the reconstructed image and the perceptible image of the reconstructed image to obtain a defect detection result.

[0007] According to another aspect of the present invention, there is provided a neural network training device, comprising: an input unit, configured to input a training image and calculate a perceptible map of the training image; a generation unit, configured to generate a training defect image at least based on the training image and the perceptible map of the training image, wherein the training defect image comprises one or more defects generated based on the perceptible map of the training image; a reconstruction unit, configured to reconstruct the training defect image through a reconstruction network including a first submodule to obtain a reconstructed training reconstructed image with defects removed and a perceptible map of the training reconstructed image, wherein the first submodule is used to screen and process features by screening feature channels and extracting spatial relationships between the channels; and a training unit, configured to perform defect detection based on the training image, the training reconstructed image and the perceptible map of the training reconstructed image, so as to train the neural network and adjust the parameters of the neural network.

[0008] According to another aspect of the present invention, there is provided a neural network training device, comprising: a processor; and a memory, wherein computer program instructions are stored in the memory, wherein when the computer program instructions are executed by the processor, the processor is caused to perform the following steps: input a training image, and calculate a perceptible map of the training image; generate a training defect image at least based on the training image and the perceptible map of the training image, wherein the training defect image includes one or more defects generated based on the perceptible map of the training image; reconstruct the training defect image by using a reconstruction network including a first submodule to obtain a reconstructed training reconstructed image with defects removed after reconstruction and a perceptible map of the training reconstructed image, wherein the first submodule is used to screen and process features by screening feature channels and extracting spatial relationships between the channels; and perform defect detection based on the training image, the training reconstructed image and the perceptible map of the training reconstructed image to train the neural network and adjust the parameters of the neural network.

[0009] According to another aspect of the present invention, there is provided a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the following steps: input a training image, and calculate a perceptible map of the training image; generate a training defect image at least based on the training image and the perceptible map of the training image, wherein the training defect image includes one or more defects generated based on the perceptible map of the training image; reconstruct the training defect image by using a reconstruction network including a first submodule to obtain a reconstructed training reconstructed image with defects removed and a perceptible map of the training reconstructed image, wherein the first submodule is used to screen and process features by screening feature channels and extracting spatial relationships between the channels; and perform defect detection based on the training image, the training reconstructed image, and the perceptible map of the training reconstructed image to train the neural network and adjust the parameters of the neural network.

[0010] According to another aspect of the present invention, there is provided an apparatus for defect detection using a neural network, comprising: an input unit, configured to input an image to be detected, reconstruct the image to be detected by a reconstruction network including a first submodule, and obtain a reconstructed image with defects removed after reconstruction and a perceptible image of the reconstructed image, wherein the first submodule is used to screen and process features by screening feature channels and extracting spatial relationships between each channel; a detection unit, configured to perform defect segmentation according to the image to be detected, the reconstructed image and the perceptible image of the reconstructed image, and obtain a defect detection result.

[0011] According to another aspect of the present invention, there is provided an apparatus for defect detection using a neural network, comprising: a processor; and a memory, wherein computer program instructions are stored in the memory, wherein when the computer program instructions are executed by the processor, the processor performs the following steps: inputting an image to be detected, reconstructing the image to be detected through a reconstruction network including a first submodule, and obtaining a reconstructed image with defects removed after reconstruction and a perceptible image of the reconstructed image, wherein the first submodule is used to screen and process features by screening feature channels and extracting spatial relationships between each channel; and performing defect segmentation according to the image to be detected, the reconstructed image and the perceptible image of the reconstructed image to obtain a defect detection result.

[0012] According to another aspect of the present invention, there is provided a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the following steps: input an image to be detected, reconstruct the image to be detected by a reconstruction network including a first submodule, and obtain a reconstructed image with defects removed after reconstruction and a perceptible image of the reconstructed image, wherein the first submodule is used to screen and process features by screening feature channels and extracting spatial relationships between the channels; and perform defect segmentation according to the image to be detected, the reconstructed image and the perceptible image of the reconstructed image to obtain a defect detection result.

[0013] According to the above-mentioned neural network training method, device and computer-readable storage medium, as well as the method, device and computer-readable storage medium for defect detection using a neural network of the present invention, it is possible to improve the training method of the neural network and improve the accuracy of defect detection by introducing a first submodule for screening and processing features by screening feature channels and extracting the spatial relationship of each channel, or further introducing a second submodule for extracting the difference between the training image and the training reconstructed image and the perceptible image of the training reconstructed image, thereby achieving accurate generation and robust detection of disordered defects for different types of products or objects, avoiding the collection and manual labeling process of a large amount of defect data, and improving user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other objects, features and advantages of the present invention will become more apparent by describing in detail the embodiments of the present invention in conjunction with the accompanying drawings.

[0015] Figure 1 A flowchart showing a neural network training method according to an embodiment of the present invention is shown;

[0016] Figure 2 An example of a defect mask image according to an embodiment of the present invention is shown;

[0017] Figure 3 shows an example of a defect control panel according to an embodiment of the present invention;

[0018] Figure 4 A schematic diagram of the structure of a reconstructed network according to an embodiment of the present invention is shown;

[0019] Figure 5 A schematic diagram of the structure of a segmentation network according to an embodiment of the present invention is shown;

[0020] Figure 6 A flow chart showing a method for defect detection using a neural network according to an embodiment of the present invention;

[0021] Figure 7A block diagram showing a neural network training device according to an embodiment of the present invention;

[0022] Figure 8 A block diagram showing a neural network training device according to an embodiment of the present invention;

[0023] Fig. 9 A block diagram showing an apparatus for defect detection using a neural network according to an embodiment of the present invention;

[0024] Fig.10 A block diagram of a device for defect detection using a neural network according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0025] The following will describe a neural network training method, apparatus, and computer-readable storage medium according to an embodiment of the present invention, as well as a method, apparatus, and computer-readable storage medium for defect detection using a neural network, with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same elements throughout. It should be understood that the embodiments described herein are merely illustrative and should not be interpreted as limiting the scope of the present invention.

[0026] When training a neural network for defect detection, it is often necessary to introduce a large number of training images containing various defects as samples to adjust the parameters of the neural network. However, in the actual training process, it is often necessary to train a corresponding model for each type of product or object. As the number of product types increases, the model training time and storage cost also increase significantly. The embodiment of the present invention provides a unified neural network training and defect detection method, which does not require manual annotation and only uses one model to improve the training method of the neural network and can robustly detect defects of multiple products or objects. It should be noted that the method and device provided by the embodiment of the present invention are not limited to analysis scenarios for products such as cables, tablets, and wooden boards, but can also include other computer vision tasks, such as defect detection on the surface of a road surface, a chip surface, or any other object. In order to illustrate the application of the present invention, we only list some features that are easy for readers to understand, such as color, shape, etc. However, it does not mean that the present invention can only extract and rely on these features. Optionally, the embodiment of the present invention can be implemented based on a convolutional neural network (CNN), which can extract more complex features at multiple semantic levels instead of only extracting limited low-level features.

[0027] Figure 1 FIG. 1 is a flowchart of a neural network training method 100 according to an embodiment of the present invention. Figure 1 A neural network training method according to an embodiment of the present invention is described.

[0028] In step S101 , a training image is input and a perceptible map of the training image is calculated.

[0029] In this step, the input training images used to train the neural network can be images of products or objects without defects. In order to enrich the training samples as much as possible, the training images can be the same product or object captured at different angles, different lighting conditions, and different backgrounds to further improve the defect detection accuracy of the trained neural network.

[0030] After inputting the training image, the noticeable map of the training image can be calculated using the input training image, and the calculated noticeable map of the training image can be a multi-scale noticeable map. The multi-scale noticeable map of the training image can reflect the change in the strength of each image feature in the training image, that is, it reflects the difference in the visual significance of each image feature in the training image. The multi-scale noticeable map of the training image can make the generated training defect image more natural and realistic, and can also make the subsequent defect detection and positioning process more accurate.

[0031] In one example, a multi-scale just-noticeable map of the training image can be calculated using an input training image. For example, the just-noticeable map of the training image can be calculated based on a just-noticeable distortion (JND) model. Among them, the JND value is an important indicator of visual saliency, which describes the degree of human perception of image quality. The JND value reveals the perceptual threshold of image intensity changes that can be noticed by the human visual system. Any perceptible distortion level must be greater than the JND value. In an embodiment of the present invention, a training defect image sample set with diversity and multiple scale samples can be generated with the help of a JND model, and can be used in subsequent defect detection and positioning processes.

[0032] Optionally, the process of calculating the multi-scale minimum perceptible map of the training image according to the minimum perceptible error model may include: first calculating the minimum perceptible map of the training image according to the minimum perceptible error model, processing the minimum perceptible map of the training image using multiple coefficients (for example, multiplying the multiple coefficients by the minimum perceptible map of the training image respectively), thereby obtaining the multi-scale minimum perceptible map of the training image. For example, three coefficients (such as 0.33, 0.66 and 1) may be selected to be multiplied by the minimum perceptible map of the training image respectively, thereby obtaining the minimum perceptible maps of three scales of the training image corresponding to three different scales; for another example, four coefficients (such as 0.25, 1, 0.75 and 1) may be selected to be multiplied by the minimum perceptible map of the training image respectively, thereby obtaining the minimum perceptible maps of four scales of the training image corresponding to four different scales. The above generation method of the multi-scale minimum perceptible map of the training image is only an example. In the embodiment of the present invention, various methods can be used to generate multi-scale minimum perceptible maps of training images at various scales according to the requirements of different application scenarios. As mentioned above, the multi-scale minimum perceptible map of the training image can reflect the changes in the range and strength of each image feature in the training image. On the basis of human perception, the distribution and changes of the grayscale values ​​therein reflect the difference in the visual significance of each image feature in the training image.

[0033] In step S102, a training defect image is generated at least according to the training image and the perceptible map of the training image, wherein the training defect image includes one or more defects generated based on the perceptible map of the training image.

[0034] In this step, the process of generating a training defect image may first include generating a defect mask image, and generating one or more defect texture images based on the training image; then, based on a defect control panel and the defect mask image, and using a multi-scale perceptible map of the training image, the training image and the one or more defect texture images may be fused to generate a training defect image, wherein the defect control panel is used to control the proportion of the defect texture image occupying the training image.

[0035] Optionally, the defect mask image can be generated by selecting one or more defect areas of different sizes and shapes in the training image. In addition, optionally, the defect texture image can be randomly generated based on any image, or it can be generated by processing the training image. In one example, in order to enhance the realistic effect of the simulated defect, the defect texture image can be generated by using the result of enhancing the training image. For example, the training image can be subjected to various random processing including flipping, scaling, rotation, region cropping, color conversion, brightness change, exposure change, etc. to obtain a defect texture image with various textures. Figure 2An example of a defect mask image according to an embodiment of the present invention is shown. Specifically, Figure 2 The multi-scale defect mask image marked with defect areas of different sizes is generated by selecting one or more defect areas of different sizes and shapes in the training image.

[0036] After obtaining the defect mask image and the defect texture image, optionally, the training image and one or more defect texture images can be fused according to the defect control panel combined with the defect mask image, using the multi-scale perceptible map of the training image as a weight map to generate a training defect image with various simulated defects. According to the corresponding principle of the perceptible map, the perceptible map of the training image can reveal the area with high color contrast and saturation in the training image, and the area with a larger grayscale in the perceptible map can occupy more defect components in the training defect image, that is, more weight can be given to the defect texture image. Conversely, the area with a smaller grayscale in the perceptible map of the training image can give a relatively more weight to the input training image. On this basis, the application of the multi-scale perceptible map of the training image can obtain fused training defect images at various scales, so that the obtained training defect image is more realistic and further used in the subsequent defect detection process.

[0037] Figure 3 An example diagram of various defect control panels according to an embodiment of the present invention is shown. Figure 3 The defect control panel shown in can be used to limit the area occupied by the defect texture image when the defect texture image is merged with the training image. When in use, the defect control panel used can be randomly selected from multiple defect control panels, or the defect control panel used can be selected according to a preset order or standard, which is not limited here. Optionally, the defect texture image can occupy an area with a width of n% and a height of m% at a certain position (e.g., above, below, left or right) in the training image, where m and n can be randomly selected from a preset interval, for example, the preset interval can be [50,80]. In addition, the defect control panel can also be, for example, an area covering the entire training image, which is also not limited here. Of course, the above-mentioned area setting and size selection of the defect control panel are only examples. In actual applications, any control method of the defect control panel can be selected according to the application scenario, which will not be repeated here. By selecting and using the defect control panel, the proportion of the defect texture image occupying the entire area of ​​the training image can be controlled during the generation process of the training defect image, so that the generation result of the training defect image is more diversified and randomized, so as to obtain more accurate neural network training and defect detection results.

[0038] Optionally, when a multi-scale perceptible map of a training image is generated, a multi-scale defect mask image can be generated accordingly, so as to fuse the training image and the defect texture image accordingly, and finally obtain a training defect image. Specifically, the multi-scale defect mask image can be a defect mask image with defect areas of different sizes marked corresponding to each scale in the multi-scale perceptible map of the training image. For example, the generation process of the multi-scale minimum perceptible map of the aforementioned training image can be referred to, and the initially generated defect mask image can also be processed accordingly using multiple coefficients (for example, the multiple coefficients are multiplied by the defect mask image respectively), to obtain a corresponding multi-scale defect mask image, so as to define the range of different defect areas at different scales. In the subsequent defect generation process, based on the defect control panel, in the process of fusing the training image and the defect texture image using these multi-scale defect mask images, the defect areas marked by the multi-scale defect mask image can be subjected to corresponding image processing, for example, each defect area of ​​the multi-scale can be processed in the fusion process by means of Gaussian blur, so as to obtain a more realistic and natural training defect image by fusion.

[0039] The above specific method of generating training defect images is only an example. In actual applications, training defect images can be generated in various ways according to corresponding scenarios, and no limitation is made here.

[0040] In step S103, the training defect image is reconstructed by a reconstruction network including a first submodule to obtain a reconstructed training reconstructed image with defects removed and a perceptible image of the training reconstructed image, wherein the first submodule is used to screen and process features by screening feature channels and extracting spatial relationships between channels.

[0041] Figure 4 FIG. 1 shows a schematic diagram of a structure of a reconstructed network according to an embodiment of the present invention. Figure 4In the figure, the training defect image passes through a reconstruction network including an encoder and a decoder to generate a training reconstructed image after defect removal and a perceptible map (or minimum perceptible map) of the training reconstructed image. As shown in the figure, in each layer of the encoder and decoder of the reconstruction network, a basic submodule and a first submodule may be respectively provided, and the basic submodule and the first submodule are interconnected at each layer. Optionally, the basic submodule may include two consecutive 3x3 convolutional layers, a normalization layer, and a ReLU activation layer to extract features at each level. The first submodule may include a channel attention subblock and a spatial attention subblock. Optionally, the channel attention subblock may extract the relationship between different feature channels through an average pooling layer and two consecutive 1x1 convolutional layers, and add the relationship between different feature channels back to the training defect image to screen out relatively important feature channels. The spatial attention sub-block can further extract the spatial relationship of each feature channel. The extracted spatial relationship can be cascaded together after passing through the average pooling layer and the maximum pooling layer, and then passed through the standard convolution layer and the hole convolution network (such as the 3x3 convolution layer with holes) to screen the important feature areas. The existence of the hole convolution network can expand the feature receptive field while maintaining the spatial resolution.

[0042] After the input training defect image enters the reconstruction network, the encoder can be used to extract the input features first. Optionally, the convolution, normalization and pooling processes of different parameters of the encoder in the reconstruction network can be used to extract features at different levels: first, the input training defect image can be convolved with a convolution kernel to obtain a convolution map; then, the convolution map can be normalized using a linear correction unit and a batch normalization method to obtain a normalized convolution map; then, the normalized convolution map can be applied with a maximum or average pooling process. In order to obtain rich multi-scale features, the encoder can adjust the relevant parameters to repeat the above process multiple times to extract the corresponding multi-scale feature map through multiple downsampling processes. After being processed by the encoder of the reconstruction network, the decoder can also be used to restore the resolution of the multi-scale feature map through convolution, normalization and upsampling processes. The reconstruction network combines the features of the encoder and decoder with the same resolution and uses multiple sets of convolution processes. During the training phase of the neural network, the reconstruction network can learn and update various parameters and weights of the neural network under the supervision of the input training image and the perceptible map of the training image, thereby converting the input training defect image into the corresponding training reconstruction image and the perceptible map of the corresponding training reconstruction image.

[0043] In step S104, defect detection is performed based on the training image, the training reconstructed image and the perceptible map of the training reconstructed image to train the neural network and adjust the parameters of the neural network.

[0044] Optionally, performing defect detection at least based on the training image, the training reconstructed image and the perceptible image of the training reconstructed image may include: extracting the differences between the training image and the training reconstructed image, between the training image and the perceptible image of the training reconstructed image, and between the training image and the training reconstructed image and the perceptible image of the training reconstructed image, respectively, through a segmentation network including a second submodule, to perform defect detection.

[0045] Figure 5 FIG. 4 shows a schematic diagram of the structure of a segmentation network according to an embodiment of the present invention. Figure 5 The segmentation network in may include a second submodule, and an encoder and a decoder connected to the second submodule. Figure 5 In , by inputting a training image, a training reconstructed image, and a perceptible map of the training reconstructed image, a second submodule is used to extract difference features between the multi-level training images, the training reconstructed image, and the perceptible map of the training reconstructed image. First, the second submodule can be used to first extract the differences between the training image and the training reconstructed image, between the training image and the perceptible map of the training reconstructed image, and between the training image and the training reconstructed image and the perceptible map of the training reconstructed image, and then the above differences are combined in series to form a difference feature description whose number of feature channels is equal to the sum of the feature channels of these images, and it is sent to a network including an encoder and a decoder for defect segmentation to obtain the result of defect detection and locate the defect. In Figure 5 In the above example, the specific network structure of the encoder and decoder is the same as Figure 4 Similar to the above, each level of the encoder and the decoder may also be provided with a first submodule, which will not be described in detail here.

[0046] pass Figure 5 The defect segmentation and detection of the segmentation network can locate the defects in the training images and obtain the judgment or probability representation of the existence of defects.

[0047] Optionally, when locating defects, the segmentation network may classify and represent the pixel values ​​in the image obtained to represent the defects in a segmented manner, in order to more intuitively reflect the location of the defects. For example, when outputting an image to represent a defect, the pixels in its grayscale image may be divided into three parts according to their grayscale values, for example, represented by normalized results of 0, 0.5 and 1, respectively, in order to more clearly reflect the location of the defect. Of course, the above-mentioned process of segmenting and classifying defects is only an example. In practical applications, various pixel value classification and normalization operations may be performed on the image used to represent the defect as needed, which is not limited here. After locating and representing the defect, the segmentation network may output a judgment on whether there is a defect in the training image, such as whether the training image is a defective image or not, as the result of defect detection.

[0048] According to the above-mentioned neural network training method of the embodiment of the present invention, it is possible to improve the training method of the neural network and improve the accuracy of defect detection by introducing a first submodule for screening and processing features by screening feature channels and extracting the spatial relationship of each channel, or further introducing a second submodule for extracting the difference between the training image and the training reconstructed image and the perceptible image of the training reconstructed image, thereby realizing accurate generation and robust detection of disordered defects for different types of products or objects, avoiding the collection and manual labeling process of a large amount of defect data, and improving user experience.

[0049] In addition, according to another embodiment of the present invention, the neural network training is not limited to the specific operation process of the above-mentioned neural network training method. Furthermore, the architecture and process of the neural network can also be used to achieve other requirements in various application scenarios. For example, the architecture of the above-mentioned neural network can be used to realize diversified samples, such as the production of human faces, animated images, etc.; or it can be applied to other corresponding scenarios that need to introduce image defects or noise to introduce various relatively realistic simulated defects or noise to the image. The above method according to the embodiment of the present invention can combine training images and introduce various noises to obtain diversified samples to be applicable to a variety of application scenarios.

[0050] Figure 6 FIG. 6 is a flow chart of a method 600 for defect detection using a neural network according to an embodiment of the present invention. Figure 1 The process shown below trains the neural network for defect detection. Figure 6 A method for defect detection using a neural network according to an embodiment of the present invention is described.

[0051] In step S601, an image to be detected is input, and the image to be detected is reconstructed by a reconstruction network including a first submodule to obtain a reconstructed image with defects removed and a perceptible image of the reconstructed image, wherein the first submodule is used to screen and process features by screening feature channels and extracting spatial relationships between each channel.

[0052] In this step, the image to be detected can be input into Figure 1 The reconstruction network of the first submodule in the trained neural network shown in the process is reconstructed. Optionally, the image to be detected can be an image without defects or an image with one or more defects. In the process of detecting the image to be detected, a perceptible image of the reconstructed image can be obtained while obtaining the reconstructed image with defects removed, or a multi-scale perceptible image of the reconstructed image can be further obtained for subsequent defect detection and positioning processes. The specific structure of the reconstruction network can be as follows: Figure 4 As shown, no further description is given here.

[0053] After the input image to be detected enters the reconstruction network, the encoder can be used to extract the input features first. Optionally, the convolution, normalization and pooling processes of different parameters of the encoder in the reconstruction network can be used to extract features at different levels: first, the input image to be detected can be convolved with a convolution kernel to obtain a convolution map; then, the convolution map can be normalized by a linear correction unit and a batch normalization method to obtain a normalized convolution map; then, the normalized convolution map can be applied to the maximum or average pooling process. In order to obtain rich multi-scale features, the encoder can adjust relevant parameters to repeat the above process multiple times to extract the corresponding multi-scale feature map through multiple downsampling processes. After being processed by the encoder of the reconstruction network, the decoder can also be used to restore the resolution of the multi-scale feature map through convolution, normalization and upsampling processes. The reconstruction network combines the features of the encoder and the decoder with the same resolution, and uses multiple sets of convolution processes to convert the input image to be detected into a corresponding reconstructed image with defects removed and a perceptible map of the corresponding reconstructed image.

[0054] In step S602, defect segmentation is performed according to the image to be detected, the reconstructed image and the perceptible image of the reconstructed image to obtain a defect detection result.

[0055] Optionally, performing defect segmentation at least based on the image to be detected, the reconstructed image and the perceptible image of the reconstructed image may include: extracting the differences between the image to be detected and the reconstructed image, between the perceptible image of the image to be detected and the reconstructed image, and between the image to be detected and the reconstructed image and the perceptible image of the reconstructed image, respectively, through a segmentation network including a second submodule, so as to perform defect detection. The specific structure of the segmentation network may be as follows: Figure 5 As shown, no further description is given here.

[0056] pass Figure 5 The defect segmentation and detection of the segmentation network can locate the defects in the image to be inspected and obtain the judgment or probability representation of the existence of the defects.

[0057] Optionally, when locating defects, the segmentation network may classify and represent the pixel values ​​in the image obtained to represent the defects in segments, in order to more intuitively reflect the location of the defects. For example, when outputting an image to represent a defect, the pixels in its grayscale image may be divided into three parts according to their grayscale values, for example, represented by normalized results of 0, 0.5 and 1, respectively, in order to more clearly reflect the location of the defect. Of course, the above-mentioned process of segmenting and classifying defects is only an example. In practical applications, various pixel value classification and normalization operations may be performed on the image used to represent the defect as needed, which is not limited here. After locating and representing the defect, the segmentation network may output a judgment on whether there is a defect in the image to be detected, such as whether the image to be detected is a defective image or not, as the result of defect detection.

[0058] According to the above-mentioned method for defect detection using a neural network in an embodiment of the present invention, the accuracy of defect detection can be improved by introducing a first submodule for screening and processing features by screening feature channels and extracting spatial relationships between each channel, or further introducing a second submodule for extracting the difference between the training image and the training reconstructed image and the perceptible image of the training reconstructed image, thereby achieving accurate generation and robust detection of disordered defects for different types of products or objects, avoiding the collection and manual labeling process of a large amount of defect data, and improving user experience.

[0059] Below, refer to Figure 7 The neural network training device according to an embodiment of the present invention is described. Figure 7 FIG. 7 is a block diagram of a neural network training device 700 according to an embodiment of the present invention. Figure 7As shown, the neural network training device 700 includes an input unit 710, a generation unit 720, a reconstruction unit 730, and a training unit 740. In addition to these units, the neural network training device 700 may also include other components. However, since these components are irrelevant to the content of the embodiment of the present invention, their illustration and description are omitted here. In addition, since the specific details of the following operations performed by the neural network training device 700 according to the embodiment of the present invention are the same as those described above with reference to Figure 1 The details described are the same, so repeated description of the same details is omitted here to avoid repetition.

[0060] Figure 7 The input unit 710 of the neural network training device 700 inputs a training image and calculates a perceptible map of the training image.

[0061] The input unit 710 inputs a training image and calculates a perceptual map of the training image.

[0062] The training images input by the input unit 710 for training the neural network may be images of products or objects without defects. In order to enrich the training samples as much as possible, the training images may be images of the same product or object captured at different angles, different lighting conditions, and different backgrounds, so as to further improve the defect detection accuracy of the trained neural network.

[0063] After inputting the training image, the noticeable map of the training image can be calculated using the input training image, and the calculated noticeable map of the training image can be a multi-scale noticeable map. The multi-scale noticeable map of the training image can reflect the change in the strength of each image feature in the training image, that is, it reflects the difference in the visual significance of each image feature in the training image. The multi-scale noticeable map of the training image can make the generated training defect image more natural and realistic, and can also make the subsequent defect detection and positioning process more accurate.

[0064] In one example, a multi-scale just-noticeable map of the training image can be calculated using an input training image. For example, the just-noticeable map of the training image can be calculated based on a just-noticeable distortion (JND) model. Among them, the JND value is an important indicator of visual saliency, which describes the degree of human perception of image quality. The JND value reveals the perceptual threshold of image intensity changes that can be noticed by the human visual system. Any perceptible distortion level must be greater than the JND value. In an embodiment of the present invention, a training defect image sample set with diversity and multiple scale samples can be generated with the help of a JND model, and can be used in subsequent defect detection and positioning processes.

[0065] Optionally, the process of calculating the multi-scale minimum perceptible map of the training image according to the minimum perceptible error model may include: first calculating the minimum perceptible map of the training image according to the minimum perceptible error model, processing the minimum perceptible map of the training image using multiple coefficients (for example, multiplying the multiple coefficients by the minimum perceptible map of the training image respectively), thereby obtaining the multi-scale minimum perceptible map of the training image. For example, three coefficients (such as 0.33, 0.66 and 1) may be selected to be multiplied by the minimum perceptible map of the training image respectively, thereby obtaining the minimum perceptible maps of three scales of the training image corresponding to three different scales; for another example, four coefficients (such as 0.25, 1, 0.75 and 1) may be selected to be multiplied by the minimum perceptible map of the training image respectively, thereby obtaining the minimum perceptible maps of four scales of the training image corresponding to four different scales. The above generation method of the multi-scale minimum perceptible map of the training image is only an example. In the embodiment of the present invention, various methods can be used to generate multi-scale minimum perceptible maps of training images at various scales according to the requirements of different application scenarios. As mentioned above, the multi-scale minimum perceptible map of the training image can reflect the changes in the range and strength of each image feature in the training image. On the basis of human perception, the distribution and changes of the grayscale values ​​therein reflect the difference in the visual significance of each image feature in the training image.

[0066] The generating unit 720 generates a training defect image according to at least the training image and the perceptible map of the training image, wherein the training defect image includes one or more defects generated based on the perceptible map of the training image.

[0067] The process of generating a training defect image by the generation unit 720 may first include generating a defect mask image, and generating one or more defect texture images based on the training image; then, the training image and the one or more defect texture images may be fused according to a defect control panel and the defect mask image, and using a multi-scale perceptible map of the training image, to generate a training defect image, wherein the defect control panel is used to control the proportion of the defect texture image occupying the training image.

[0068] Optionally, the defect mask image can be generated by selecting one or more defect areas of different sizes and shapes in the training image. In addition, optionally, the defect texture image can be randomly generated based on any image, or it can be generated by processing the training image. In one example, in order to enhance the realistic effect of the simulated defect, the defect texture image can be generated by using the result of enhancing the training image. For example, the training image can be subjected to various random processing including flipping, scaling, rotation, region cropping, color conversion, brightness change, exposure change, etc. to obtain a defect texture image with various textures. Figure 2 An example of a defect mask image according to an embodiment of the present invention is shown. Specifically, Figure 2 The multi-scale defect mask image marked with defect areas of different sizes is generated by selecting one or more defect areas of different sizes and shapes in the training image.

[0069] After obtaining the defect mask image and the defect texture image, optionally, the training image and one or more defect texture images can be fused according to the defect control panel combined with the defect mask image, using the multi-scale perceptible map of the training image as a weight map to generate a training defect image with various simulated defects. According to the corresponding principle of the perceptible map, the perceptible map of the training image can reveal the area with high color contrast and saturation in the training image, and the area with a larger grayscale in the perceptible map can occupy more defect components in the training defect image, that is, more weight can be given to the defect texture image. Conversely, the area with a smaller grayscale in the perceptible map of the training image can give a relatively more weight to the input training image. On this basis, the application of the multi-scale perceptible map of the training image can obtain fused training defect images at various scales, so that the obtained training defect image is more realistic and further used in the subsequent defect detection process.

[0070] Figure 3 An example diagram of various defect control panels according to an embodiment of the present invention is shown. Figure 3 The defect control panel shown in can be used to limit the area occupied by the defect texture image when the defect texture image is merged with the training image. When in use, the defect control panel used can be randomly selected from multiple defect control panels, or the defect control panel used can be selected according to a preset order or standard, which is not limited here. Optionally, the defect texture image can occupy an area with a width of n% and a height of m% at a certain position (e.g., above, below, left or right) in the training image, where m and n can be randomly selected from a preset interval, for example, the preset interval can be [50,80]. In addition, the defect control panel can also be, for example, an area covering the entire training image, which is also not limited here. Of course, the above-mentioned area setting and size selection of the defect control panel are only examples. In actual applications, any control method of the defect control panel can be selected according to the application scenario, which will not be repeated here. By selecting and using the defect control panel, the proportion of the defect texture image occupying the entire area of ​​the training image can be controlled during the generation process of the training defect image, so that the generation result of the training defect image is more diversified and randomized, so as to obtain more accurate neural network training and defect detection results.

[0071] Optionally, when a multi-scale perceptible map of a training image is generated, a multi-scale defect mask image can be generated accordingly, so as to fuse the training image and the defect texture image accordingly, and finally obtain a training defect image. Specifically, the multi-scale defect mask image can be a defect mask image with defect areas of different sizes marked corresponding to each scale in the multi-scale perceptible map of the training image. For example, the generation process of the multi-scale minimum perceptible map of the aforementioned training image can be referred to, and the initially generated defect mask image can also be processed accordingly using multiple coefficients (for example, the multiple coefficients are multiplied by the defect mask image respectively), to obtain a corresponding multi-scale defect mask image, so as to define the range of different defect areas at different scales. In the subsequent defect generation process, based on the defect control panel, in the process of fusing the training image and the defect texture image using these multi-scale defect mask images, the defect areas marked by the multi-scale defect mask image can be subjected to corresponding image processing, for example, each defect area of ​​the multi-scale can be processed in the fusion process by means of Gaussian blur, so as to obtain a more realistic and natural training defect image by fusion.

[0072] The above specific method of generating training defect images is only an example. In actual applications, training defect images can be generated in various ways according to corresponding scenarios, and no limitation is made here.

[0073] The reconstruction unit 730 reconstructs the training defect image through a reconstruction network including a first submodule to obtain a reconstructed training reconstructed image with defects removed and a perceptible image of the training reconstructed image, wherein the first submodule is used to screen and process features by screening feature channels and extracting spatial relationships between each channel.

[0074] Figure 4 FIG. 1 shows a schematic diagram of a structure of a reconstructed network according to an embodiment of the present invention. Figure 4In the figure, the training defect image passes through a reconstruction network including an encoder and a decoder to generate a training reconstructed image after defect removal and a perceptible map (or minimum perceptible map) of the training reconstructed image. As shown in the figure, in each layer of the encoder and decoder of the reconstruction network, a basic submodule and a first submodule may be respectively provided, and the basic submodule and the first submodule are interconnected at each layer. Optionally, the basic submodule may include two consecutive 3x3 convolutional layers, a normalization layer, and a ReLU activation layer to extract features at each level. The first submodule may include a channel attention subblock and a spatial attention subblock. Optionally, the channel attention subblock may extract the relationship between different feature channels through an average pooling layer and two consecutive 1x1 convolutional layers, and add the relationship between different feature channels back to the training defect image to screen out relatively important feature channels. The spatial attention sub-block can further extract the spatial relationship of each feature channel. The extracted spatial relationship can be cascaded together after passing through the average pooling layer and the maximum pooling layer, and then passed through the standard convolution layer and the hole convolution network (such as the 3x3 convolution layer with holes) to screen the important feature areas. The existence of the hole convolution network can expand the feature receptive field while maintaining the spatial resolution.

[0075] After the input training defect image enters the reconstruction network, the encoder can be used to extract the input features first. Optionally, the convolution, normalization and pooling processes of different parameters of the encoder in the reconstruction network can be used to extract features at different levels: first, the input training defect image can be convolved with a convolution kernel to obtain a convolution map; then, the convolution map can be normalized using a linear correction unit and a batch normalization method to obtain a normalized convolution map; then, the normalized convolution map can be applied with a maximum or average pooling process. In order to obtain rich multi-scale features, the encoder can adjust the relevant parameters to repeat the above process multiple times to extract the corresponding multi-scale feature map through multiple downsampling processes. After being processed by the encoder of the reconstruction network, the decoder can also be used to restore the resolution of the multi-scale feature map through convolution, normalization and upsampling processes. The reconstruction network combines the features of the encoder and decoder with the same resolution and uses multiple sets of convolution processes. During the training phase of the neural network, the reconstruction network can learn and update various parameters and weights of the neural network under the supervision of the input training image and the perceptible map of the training image, thereby converting the input training defect image into the corresponding training reconstruction image and the perceptible map of the corresponding training reconstruction image.

[0076] The training unit 740 performs defect detection according to the training image, the training reconstructed image and the perceptible map of the training reconstructed image to train the neural network and adjust the parameters of the neural network.

[0077] Optionally, the training unit 740 extracts the differences between the training image and the training reconstructed image, between the perceptible images of the training image and the training reconstructed image, and between the training image and the training reconstructed image and the perceptible images of the training reconstructed image through a segmentation network including a second submodule to perform defect detection.

[0078] Figure 5 FIG. 4 shows a schematic diagram of the structure of a segmentation network according to an embodiment of the present invention. Figure 5 The segmentation network in may include a second submodule, and an encoder and a decoder connected to the second submodule. Figure 5 In , by inputting a training image, a training reconstructed image, and a perceptible map of the training reconstructed image, a second submodule is used to extract difference features between the multi-level training images, the training reconstructed image, and the perceptible map of the training reconstructed image. First, the second submodule can be used to first extract the differences between the training image and the training reconstructed image, between the training image and the perceptible map of the training reconstructed image, and between the training image and the training reconstructed image and the perceptible map of the training reconstructed image, and then the above differences are combined in series to form a difference feature description whose number of feature channels is equal to the sum of the feature channels of these images, and it is sent to a network including an encoder and a decoder for defect segmentation to obtain the result of defect detection and locate the defect. In Figure 5 In the above example, the specific network structure of the encoder and decoder is the same as Figure 4 Similar to the above, each level of the encoder and the decoder may also be provided with a first submodule, which will not be described in detail here.

[0079] pass Figure 5 The defect segmentation and detection of the segmentation network can locate the defects in the training images and obtain the judgment or probability representation of the existence of defects.

[0080] Optionally, when locating defects, the segmentation network may classify and represent the pixel values ​​in the image obtained to represent the defects in a segmented manner, in order to more intuitively reflect the location of the defects. For example, when outputting an image to represent a defect, the pixels in its grayscale image may be divided into three parts according to their grayscale values, for example, represented by normalized results of 0, 0.5 and 1, respectively, in order to more clearly reflect the location of the defect. Of course, the above-mentioned process of segmenting and classifying defects is only an example. In practical applications, various pixel value classification and normalization operations may be performed on the image used to represent the defect as needed, which is not limited here. After locating and representing the defect, the segmentation network may output a judgment on whether there is a defect in the training image, such as whether the training image is a defective image or not, as the result of defect detection.

[0081] According to the above-mentioned neural network training device of an embodiment of the present invention, it is possible to improve the training method of the neural network and improve the accuracy of defect detection by introducing a first submodule for screening and processing features by screening feature channels and extracting the spatial relationship of each channel, or further introducing a second submodule for extracting the difference between the training image and the training reconstructed image and the perceptible image of the training reconstructed image, thereby realizing accurate generation and robust detection of disordered defects for different types of products or objects, avoiding the collection and manual labeling process of a large amount of defect data, and improving user experience.

[0082] Below, refer to Figure 8 The neural network training device according to an embodiment of the present invention is described. Figure 8 FIG. 8 is a block diagram of a neural network training device 800 according to an embodiment of the present invention. Figure 8 As shown, the device 800 may be a computer or a server.

[0083] like Figure 8 As shown, the neural network training device 800 includes one or more processors 810 and a memory 820. Of course, in addition to this, the neural network training device 800 may also include an input device, an output device (not shown), etc. These components may be interconnected via a bus system and / or other forms of connection mechanisms. It should be noted that Figure 8 The components and structure of the neural network training device 800 shown are merely exemplary and non-limiting. The neural network training device 800 may also have other components and structures as required.

[0084] The processor 810 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may utilize computer program instructions stored in the memory 820 to perform desired functions, which may include: inputting a training image and calculating a perceptible image of the training image; generating a training defect image at least based on the training image and the perceptible image of the training image, wherein the training defect image includes one or more defects generated based on the perceptible image of the training image; reconstructing the training defect image through a reconstruction network including a first submodule to obtain a reconstructed training reconstructed image with defects removed and a perceptible image of the training reconstructed image, wherein the first submodule is used to screen and process features by screening feature channels and extracting spatial relationships between each channel; performing defect detection based on the training image, the training reconstructed image, and the perceptible image of the training reconstructed image to train the neural network and adjust the parameters of the neural network.

[0085] The memory 820 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 810 may execute the program instructions to implement the functions of the neural network training device of the embodiment of the present invention described above and / or other desired functions, and / or may execute the neural network training method according to the embodiment of the present invention. Various applications and various data may also be stored in the computer-readable storage medium.

[0086] The following describes a computer-readable storage medium according to an embodiment of the present invention, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the following steps: input a training image, and calculate a perceptible map of the training image; generate a training defect image at least based on the training image and the perceptible map of the training image, wherein the training defect image includes one or more defects generated based on the perceptible map of the training image; reconstruct the training defect image by using a reconstruction network including a first submodule to obtain a reconstructed training reconstructed image with defects removed and a perceptible map of the training reconstructed image, wherein the first submodule is used to screen and process features by screening feature channels and extracting spatial relationships between each channel; and perform defect detection based on the training image, the training reconstructed image, and the perceptible map of the training reconstructed image to train the neural network and adjust the parameters of the neural network.

[0087] Below, refer to Fig. 9 A device for defect detection using a neural network according to an embodiment of the present invention is described. Fig. 9 FIG. 9 is a block diagram of an apparatus 900 for performing defect detection using a neural network according to an embodiment of the present invention. Fig. 9 As shown, the apparatus 900 for defect detection using a neural network includes an input unit 910 and a detection unit 920. In addition to these units, the apparatus 900 may also include other components. However, since these components are irrelevant to the content of the embodiment of the present invention, their illustration and description are omitted here. In addition, since the specific details of the following operations performed by the apparatus 900 according to the embodiment of the present invention are the same as those described above with reference to Figure 6 The details described are the same, so repeated description of the same details is omitted here to avoid repetition.

[0088] Fig. 9The input unit 910 of the device 900 for defect detection using a neural network inputs an image to be detected, and reconstructs the image to be detected through a reconstruction network including a first submodule to obtain a reconstructed image with defects removed after reconstruction and a perceptible image of the reconstructed image, wherein the first submodule is used to screen and process features by screening feature channels and extracting spatial relationships between each channel.

[0089] The input unit 910 can input the image to be detected through Figure 1 The reconstruction network of the first submodule in the trained neural network shown in the process is reconstructed. Optionally, the image to be detected can be an image without defects or an image with one or more defects. In the process of detecting the image to be detected, a perceptible image of the reconstructed image can be obtained while obtaining the reconstructed image with defects removed, or a multi-scale perceptible image of the reconstructed image can be further obtained for subsequent defect detection and positioning processes. The specific structure of the reconstruction network can be as follows: Figure 4 As shown, no further description is given here.

[0090] After the input image to be detected enters the reconstruction network, the encoder can be used to extract the input features first. Optionally, the convolution, normalization and pooling processes of different parameters of the encoder in the reconstruction network can be used to extract features at different levels: first, the input image to be detected can be convolved with a convolution kernel to obtain a convolution map; then, the convolution map can be normalized by a linear correction unit and a batch normalization method to obtain a normalized convolution map; then, the normalized convolution map can be applied to the maximum or average pooling process. In order to obtain rich multi-scale features, the encoder can adjust relevant parameters to repeat the above process multiple times to extract the corresponding multi-scale feature map through multiple downsampling processes. After being processed by the encoder of the reconstruction network, the decoder can also be used to restore the resolution of the multi-scale feature map through convolution, normalization and upsampling processes. The reconstruction network combines the features of the encoder and the decoder with the same resolution, and uses multiple sets of convolution processes to convert the input image to be detected into a corresponding reconstructed image with defects removed and a perceptible map of the corresponding reconstructed image.

[0091] The detection unit 920 performs defect segmentation according to the image to be detected, the reconstructed image and the perceptible image of the reconstructed image to obtain a defect detection result.

[0092] Optionally, the detection unit 920 extracts the differences between the image to be detected and the reconstructed image, between the perceptible images of the image to be detected and the reconstructed image, and between the image to be detected and the reconstructed image and the perceptible images of the reconstructed image through a segmentation network including a second submodule, so as to perform defect detection. The specific structure of the segmentation network can be as follows: Figure 5As shown, no further description is given here.

[0093] pass Figure 5 The defect segmentation and detection of the segmentation network can locate the defects in the image to be inspected and obtain the judgment or probability representation of the existence of the defects.

[0094] Optionally, when locating defects, the segmentation network may classify and represent the pixel values ​​in the image obtained to represent the defects in segments, in order to more intuitively reflect the location of the defects. For example, when outputting an image to represent a defect, the pixels in its grayscale image may be divided into three parts according to their grayscale values, for example, represented by normalized results of 0, 0.5 and 1, respectively, in order to more clearly reflect the location of the defect. Of course, the above-mentioned process of segmenting and classifying defects is only an example. In practical applications, various pixel value classification and normalization operations may be performed on the image used to represent the defect as needed, which is not limited here. After locating and representing the defect, the segmentation network may output a judgment on whether there is a defect in the image to be detected, such as whether the image to be detected is a defective image or not, as the result of defect detection.

[0095] According to the above-mentioned device for defect detection using a neural network in an embodiment of the present invention, the accuracy of defect detection can be improved by introducing a first submodule for screening and processing features by screening feature channels and extracting spatial relationships between each channel, or further introducing a second submodule for extracting the difference between the training image and the training reconstructed image and the perceptible image of the training reconstructed image, thereby achieving accurate generation and robust detection of disordered defects for different types of products or objects, avoiding the collection and manual labeling process of a large amount of defect data, and improving user experience.

[0096] Below, refer to Fig.10 A device for defect detection using a neural network according to an embodiment of the present invention is described. Fig.10 FIG. 9 is a block diagram of an apparatus 900 for performing defect detection using a neural network according to an embodiment of the present invention. Fig.10 As shown, the device 1000 may be a computer or a server.

[0097] like Fig.10 As shown, the device 1000 includes one or more processors 1010 and a memory 1020. Of course, in addition to this, the device 1000 may also include an input device, an output device (not shown), etc. These components may be interconnected via a bus system and / or other forms of connection mechanisms. It should be noted that Fig. 9 The components and structures of the device 1000 shown are merely exemplary and non-limiting. The device 1000 may also have other components and structures as required.

[0098] The processor 1010 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may utilize computer program instructions stored in the memory 1020 to execute desired functions, which may include: inputting an image to be detected, reconstructing the image to be detected through a reconstruction network including a first submodule, and obtaining a reconstructed image with defects removed after reconstruction and a perceptible image of the reconstructed image, wherein the first submodule is used to screen and process features by screening feature channels and extracting spatial relationships between each channel; performing defect segmentation according to the image to be detected, the reconstructed image, and the perceptible image of the reconstructed image to obtain a defect detection result.

[0099] The memory 1020 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1010 may execute the program instructions to implement the functions of the apparatus for defect detection using a neural network according to the embodiment of the present invention described above and / or other desired functions, and / or may execute the method for defect detection using a neural network according to the embodiment of the present invention. Various applications and various data may also be stored in the computer-readable storage medium.

[0100] The following describes a computer-readable storage medium according to an embodiment of the present invention, on which computer program instructions are stored, wherein the computer program instructions implement the following steps when executed by a processor: input an image to be detected, reconstruct the image to be detected through a reconstruction network including a first submodule, and obtain a reconstructed image with defects removed after reconstruction and a perceptible image of the reconstructed image, wherein the first submodule is used to screen and process features by screening feature channels and extracting spatial relationships between each channel; and perform defect segmentation according to the image to be detected, the reconstructed image, and the perceptible image of the reconstructed image to obtain a defect detection result.

[0101] Of course, the above-mentioned specific embodiments are merely examples rather than limitations, and those skilled in the art can, based on the concept of the present invention, merge and combine some steps and devices from the various embodiments described separately above to achieve the effects of the present invention. Such merged and combined embodiments are also included in the present invention, and such merges and combinations are not described one by one herein.

[0102] Note that the advantages, strengths, effects, etc. mentioned in the present invention are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present invention. In addition, the specific details of the above invention are only for the purpose of illustration and facilitating understanding, not limitation, and the above details do not limit the present invention to being implemented by adopting the above specific details.

[0103] The block diagrams of the devices, apparatuses, equipment, and systems involved in the present invention are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The word "such as" used here refers to the phrase "such as but not limited to", and can be used interchangeably with it.

[0104] The step flow charts and the above method descriptions in the present invention are only illustrative examples and are not intended to require or imply that the steps of each embodiment must be performed in the order given. As will be appreciated by those skilled in the art, the order of the steps in the above embodiments can be performed in any order. Words such as "thereafter", "then", "next", etc. are not intended to limit the order of the steps; these words are only used to guide the reader through the description of these methods. In addition, any reference to a singular element, such as using the article "a", "an", or "the", is not to be construed as limiting the element to the singular.

[0105] In addition, the steps and devices in the various embodiments of this document are not limited to being implemented in a certain embodiment. In fact, based on the concept of the present invention, relevant partial steps and partial devices in the various embodiments of this document can be combined to conceive new embodiments, and these new embodiments are also included in the scope of the present invention.

[0106] Each operation of the method described above may be performed by any suitable means capable of performing the corresponding functions. The means may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs) or processors.

[0107] The various illustrated logic blocks, modules and circuits described may be implemented or performed using a general purpose processor, digital signal processor (DSP), ASIC, field programmable gate array signal (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but as an alternative, the processor may be any commercially available processor, controller, microcontroller or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

[0108] The steps of the method or algorithm described in conjunction with the present invention can be directly embedded in hardware, in a software module executed by a processor, or in a combination of the two. A software module can exist in any form of tangible storage medium. Some examples of storage media that can be used include random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, etc. A storage medium can be coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative manner, the storage medium can be integral with the processor. A software module can be a single instruction or many instructions, and can be distributed on several different code segments, between different programs, and across multiple storage media.

[0109] The method invented herein includes one or more actions for implementing the described method. The method and / or action can be interchangeable with each other without departing from the scope of the claims. In other words, unless a specific order of actions is specified, the order and / or use of specific actions can be modified without departing from the scope of the claims.

[0110] The functions described can be implemented by hardware, software, firmware or any combination thereof. If implemented in software, the functions can be stored as one or more instructions on a tangible computer-readable medium. The storage medium can be any available tangible medium that can be accessed by a computer. By way of example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device or any other tangible medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. As used herein, a disc includes a compact disc (CD), a laser disc, an optical disc, a digital versatile disc (DVD), a floppy disk and a blue disc.

[0111] Thus, a computer program product may perform the operations presented herein. For example, such a computer program product may be a computer-readable tangible medium having instructions tangibly stored (and / or encoded) thereon, which instructions may be executed by one or more processors to perform the operations described herein. The computer program product may include packaging materials.

[0112] Software or instructions may also be transmitted via a transmission medium. For example, the software may be transmitted from a website, server or other remote source using a transmission medium such as coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology such as infrared, radio or microwave.

[0113] In addition, the module and / or other appropriate means for performing the methods and techniques described herein can be downloaded and / or otherwise obtained by the user terminal and / or base station when appropriate. For example, such a device can be coupled to a server to facilitate the transmission of the means for performing the methods described herein. Alternatively, the various methods described herein can be provided via a storage component (e.g., RAM, ROM, a physical storage medium such as a CD or a floppy disk, etc.) so that the user terminal and / or base station can obtain the various methods when being coupled to the device or providing a storage component to the device. In addition, any other appropriate technology for providing the methods and techniques described herein to a device can be utilized.

[0114] Other examples and implementations are within the scope and spirit of the present invention and the appended claims. For example, due to the nature of software, the functions described above can be implemented using software executed by a processor, hardware, firmware, hard wiring, or any combination of these. Features that implement the functions can also be physically located in various locations, including being distributed so that parts of the functions are implemented at different physical locations. Moreover, as used herein, including as used in the claims, "or" used in the enumeration of items beginning with "at least one" indicates a separate enumeration, so that, for example, the enumeration of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). In addition, the wording "exemplary" does not mean that the example described is preferred or better than other examples.

[0115] Various changes, substitutions, and modifications of the techniques described herein may be made without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the present invention is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and actions described above. Currently existing or later to be developed processes, machines, manufactures, compositions of events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein may be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or actions within their scope.

[0116] The above description of the invented aspects is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features of the present invention.

[0117] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present invention to the form invented herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A neural network training method, comprising: Input a training image and calculate a perceptible map of the training image; generating a training defect image based at least on the training image and the perceptible map of the training image, the training defect image including one or more defects generated based on the perceptible map of the training image; Reconstructing the training defect image through a reconstruction network including a first submodule to obtain a reconstructed training reconstructed image with defects removed and a perceptible image of the training reconstructed image, wherein the first submodule is used to screen and process features by screening feature channels and extracting spatial relationships between each channel; Defect detection is performed based on the training image, the training reconstructed image, and the perceptible map of the training reconstructed image to train the neural network and adjust the parameters of the neural network.

2. The method of claim 1, wherein: Calculating the perceptible map of the training image comprises: A multi-scale perceptual map of the training image is calculated.

3. The method of claim 1, wherein: Generating a training defect image based at least on the training image and the perceptible map of the training image comprises: generating a defect mask image, and generating one or more defect texture images according to the training image; According to the defect control panel and the defect mask image, and using the perceptible map of the training image, the training image and the one or more defect texture images are fused to generate a training defect image, wherein the defect control panel is used to control the proportion of the defect texture image occupying the training image.

4. The method of claim 1, wherein: The first submodule includes a dilated convolutional network, which is used to expand the feature receptive field.

5. The method of claim 1, wherein: The reconstruction network includes an encoder and a decoder, and each layer of the encoder and the decoder is respectively provided with a first submodule.

6. The method of claim 1, wherein: Performing defect detection based on at least the training image, the training reconstructed image, and the perceptible map of the training reconstructed image comprises: The differences between the training image and the training reconstructed image, between the perceptible images of the training image and the training reconstructed image, and between the training image and the perceptible images of the training reconstructed image and the training image are respectively extracted through a segmentation network including a second submodule to perform defect detection.

7. The method of claim 6, wherein: The segmentation network includes an encoder and a decoder, and each level of the encoder and the decoder is respectively provided with a first submodule.

8. A defect detection method, comprising: Input an image to be detected, reconstruct the image to be detected through a reconstruction network including a first submodule, and obtain a reconstructed image with defects removed and a perceptible image of the reconstructed image, wherein the first submodule is used to screen and process features by screening feature channels and extracting spatial relationships between each channel; Defect segmentation is performed according to the image to be detected, the reconstructed image and the perceptible image of the reconstructed image to obtain a defect detection result.

9. A neural network training device, comprising: An input unit configured to input a training image and calculate a perceptible map of the training image; a generating unit configured to generate a training defect image based at least on the training image and the perceptible map of the training image, wherein the training defect image includes one or more defects generated based on the perceptible map of the training image; a reconstruction unit, configured to reconstruct the training defect image through a reconstruction network including a first submodule to obtain a reconstructed training reconstructed image with defects removed and a perceptible image of the training reconstructed image, wherein the first submodule is used to screen and process features by screening feature channels and extracting spatial relationships between each channel; A training unit is configured to perform defect detection according to the training image, the training reconstructed image and the perceptible map of the training reconstructed image to train the neural network and adjust the parameters of the neural network.

10. A neural network training device, comprising: processor; and a memory having computer program instructions stored therein, Wherein, when the computer program instructions are executed by the processor, the processor is caused to perform the following steps: Input a training image and calculate a perceptible map of the training image; generating a training defect image based at least on the training image and the perceptible map of the training image, the training defect image including one or more defects generated based on the perceptible map of the training image; Reconstructing the training defect image through a reconstruction network including a first submodule to obtain a reconstructed training reconstructed image with defects removed and a perceptible image of the training reconstructed image, wherein the first submodule is used to screen and process features by screening feature channels and extracting spatial relationships between each channel; Defect detection is performed based on the training image, the training reconstructed image, and the perceptible map of the training reconstructed image to train the neural network and adjust the parameters of the neural network.

11. A computer-readable storage medium having computer program instructions stored thereon, wherein: When the computer program instructions are executed by a processor, the following steps are implemented: Input a training image and calculate a perceptible map of the training image; generating a training defect image based at least on the training image and the perceptible map of the training image, the training defect image including one or more defects generated based on the perceptible map of the training image; Reconstructing the training defect image through a reconstruction network including a first submodule to obtain a reconstructed training reconstructed image with defects removed and a perceptible image of the training reconstructed image, wherein the first submodule is used to screen and process features by screening feature channels and extracting spatial relationships between each channel; Defect detection is performed based on the training image, the training reconstructed image, and the perceptible map of the training reconstructed image to train the neural network and adjust the parameters of the neural network.