Neural network training method and defect detection method and device as well as computer program
The neural network training method generates training defect images using perceptual maps and reconstructs images to address the challenge of detecting defects in diverse products, achieving accurate and robust detection across different types without extensive data collection or manual labeling.
Patent Information
- Application Number
- JP2024084087
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-23
- Filing Date
- 2024-05-23
- Publication Date
- 2024-12-05
- Estimated Expiration
- 2044-05-23
AI Technical Summary
Current neural network-based defect detection methods struggle to accurately and robustly detect defects in various types of products or objects, as they require separate training for each type and are not effective in handling disordered defects.
A neural network training method that involves generating training defect images using perceptual maps, reconstructing images to remove defects, and adjusting network parameters based on feature screening and spatial relationships, allowing for unified training across different types of products.
This approach enables accurate and robust defect detection for multiple types of products without the need for extensive defect data collection and manual labeling, improving user experience and detection accuracy.
Smart Images

Figure 2024169403000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to the field of image processing, and more particularly to a method for training a neural network, and a method, apparatus and computer readable storage medium for performing defect detection using a neural network. [Background technology]
[0002] Defect detection is an important factor in product quality control. Various defects occur constantly during production, and it is usually impossible to cover all the occurrences of various defects. How to detect these defects robustly and automatically is an urgent problem to be solved.
[0003] In current defect detection methods, neural networks that perform defect detection are typically trained using a large amount of mark data, and training and defect detection are performed separately for each type of product or object. However, such neural network training methods and defect detection methods using such neural networks cannot achieve accurate generation and robust detection of random defects in different types of products or objects.
[0004] Therefore, there is a need for improved neural network training methods and defect detection methods that accurately detect image defects. Summary of the Invention
[0005] In order to solve the above technical problem, a method for training a neural network according to one aspect of the present invention includes the steps of inputting a training image and calculating a perceptual map of the training image; generating a training defect image based on at least the training image and the perceptual map of the training image, where the training defect image includes one or more defects generated based on the perceptual map of the training image; performing reconstruction on the training defect image by a reconstruction network including a first submodule, and obtaining a training reconstructed image and a perceptual map of the training reconstructed image from which the defects after reconstruction have been removed, where the first submodule is used for screening and processing features by screening on feature channels and extracting spatial relationships between each channel; and performing defect detection based on the training image, the training reconstructed image, and the perceptual map of the training reconstructed image to train a neural network and adjust parameters of the neural network.
[0006] A defect detection method according to another aspect of the present invention includes a step of inputting a detected image, performing reconstruction on the detected image using a reconstruction network including a first submodule, and obtaining a reconstructed image from which defects have been removed after reconstruction and a perceptual map of the reconstructed image, wherein the first submodule is used for screening and processing features by screening feature channels and extracting spatial relationships between each channel, and a step of performing defect segmentation based on the detected image, the reconstructed image, and the perceptual map of the reconstructed image to obtain a defect detection result.
[0007] According to another aspect of the present invention, a neural network training apparatus includes an input unit for inputting a training image and calculating a perceptual map of the training image; a generation unit for generating a training defect image based on at least the training image and the perceptual map of the training image, where the training defect image includes one or more defects generated based on the perceptual map of the training image; a reconstruction unit for performing reconstruction on the training defect image using a reconstruction network including a first submodule, and obtaining a training reconstructed image and a perceptual map of the training reconstructed image from which the defects have been removed after reconstruction, where the first submodule is used for screening and processing features by screening against feature channels and extracting spatial relationships between each channel; and a training unit for performing defect detection based on the training image, the training reconstructed image, and the perceptual map of the training reconstructed image, to train a neural network and adjust parameters of the neural network.
[0008] According to another aspect of the present invention, an apparatus for training a neural network includes a processor and a memory in which computer program commands are stored, and when the computer program commands are executed by the processor, the processor is caused to execute the following steps: inputting a training image and calculating a perceptual map of the training image; generating a training defect image based on at least the training image and the perceptual map of the training image, where the training defect image includes one or more defects generated based on the perceptual map of the training image; performing reconstruction on the training defect image by a reconstruction network including a first submodule, and obtaining a training reconstructed image and a perceptual map of the training reconstructed image from which the defects have been removed after reconstruction, where the first submodule is used for screening and processing features by screening on feature channels and extracting spatial relationships between each channel; and performing defect detection based on the training image, the training reconstructed image, and the perceptual map of the training reconstructed image, to train a neural network and adjust parameters of the neural network.
[0009] According to another aspect of the present invention, a computer-readable storage medium is provided having computer program commands stored thereon, which, when executed by a processor, cause the computer to execute the following steps: inputting a training image and calculating a perceptual map of the training image; generating a training defect image based on at least the training image and the perceptual map of the training image, where the training defect image includes one or more defects generated based on the perceptual map of the training image; performing reconstruction on the training defect image by a reconstruction network including a first sub-module to obtain a training reconstructed image and a perceptual map of the training reconstructed image from which the reconstructed defects have been removed, where the first sub-module is used for screening and processing features by screening against feature channels and extracting spatial relationships between each channel; and performing defect detection based on the training image, the training reconstructed image, and the perceptual map of the training reconstructed image to train a neural network and adjust parameters of the neural network.
[0010] According to another aspect of the present invention, a defect detection device using a neural network includes an input unit for inputting a detected image, performing reconstruction on the detected image using a reconstruction network including a first submodule, and obtaining a reconstructed image from which defects have been removed after reconstruction and a perceptual map of the reconstructed image, the first submodule being used for screening and processing features by screening feature channels and extracting spatial relationships between each channel; and a detection unit for performing defect segmentation based on the detected image, the reconstructed image, and a perceptual map of the reconstructed image to obtain a defect detection result.
[0011] According to another aspect of the present invention, an apparatus for detecting defects using a neural network includes a processor and a memory in which computer program commands are stored, and when the computer program commands are executed by the processor, the processor is caused to execute the steps of: inputting a detected image, performing reconstruction on the detected image using a reconstruction network including a first submodule, and obtaining a reconstructed image from which defects have been removed after reconstruction and a perceptual map of the reconstructed image, wherein the first submodule is used for screening and processing features by screening feature channels and extracting spatial relationships between each channel, and performing defect segmentation based on the detected image, the reconstructed image, and the perceptual map of the reconstructed image to obtain a defect detection result.
[0012] According to another aspect of the present invention, a computer-readable storage medium on which computer program commands are stored causes a processor to execute the following steps when the computer program commands are executed: inputting a detected image, performing reconstruction on the detected image using a reconstruction network including a first sub-module, and obtaining a reconstructed image from which defects have been removed after reconstruction and a perceptual map of the reconstructed image, wherein the first sub-module is used for screening and processing features by screening feature channels and extracting spatial relationships between each channel; and performing defect segmentation based on the detected image, the reconstructed image, and the perceptual map of the reconstructed image to obtain a defect detection result.
[0013] Based on the above-mentioned neural network training method, apparatus, and computer-readable storage medium, and the method, apparatus, and computer-readable storage medium for performing defect detection using a neural network of the present invention, a first sub-module for screening and processing features by screening against feature channels and extracting the spatial relationship of each channel, or a second sub-module for further extracting differences between the training image, the training reconstructed image, and the perceptual map of the training reconstructed image is introduced to improve the training method of the neural network and improve the accuracy of defect detection, thereby realizing accurate generation and robust detection of disordered defects for different types of products or objects, avoiding the process of collecting a large amount of defect data and manual labeling, and improving the user experience. [Brief description of the drawings]
[0014] The above contents, objects, features and advantages of the present application will become more apparent by describing the embodiments of the present application in more detail in conjunction with the drawings. [Figure 1] 2 is a flowchart of a neural network training method according to an embodiment of the present invention. [Diagram 2] 1 is an example of a defect mask image according to an embodiment of the present invention. [Diagram 3] 1 is an illustration of a defect control panel according to an embodiment of the present invention. [Figure 4] 1 is a diagram illustrating the structure of a reconstruction network according to an embodiment of the present invention. [Diagram 5] 1 is a diagram illustrating a split network structure according to an embodiment of the present invention; [Figure 6] 4 is a flowchart of a method for detecting defects using a neural network according to an embodiment of the present invention. [Figure 7] 1 is a block diagram of a neural network training device according to an embodiment of the present invention; [Figure 8] 1 is a block diagram of a neural network training device according to an embodiment of the present invention; [Figure 9] 1 is a block diagram of an apparatus for performing defect detection using a neural network according to an embodiment of the present invention. [Figure 10] 1 is a block diagram of an apparatus for performing defect detection using a neural network according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0015] Hereinafter, a neural network training method, device, and computer readable storage medium according to the present invention, as well as a defect detection method, device, and computer readable storage medium using a neural network, will be described with reference to the accompanying drawings. In the drawings, the same numbers refer to the same elements throughout. It is apparent that the embodiments described herein are used for illustration purposes only and are not intended to limit the scope of the present invention.
[0016] When training a neural network for defect detection, it is generally necessary to introduce a large number of training images containing various defects as samples to adjust the parameters of the neural network. However, in the actual training process, it is often necessary to train a corresponding model for each type of product or object, and as the product types increase, the model training time and storage cost also increase significantly. The embodiment of the present invention provides a unified neural network training and defect detection method, which does not require manual labeling, and can improve the training method of the neural network with only one model, and can robustly detect defects of multiple types of products or objects. It should be noted that the method and apparatus provided by the embodiment of the present invention are not limited to the analysis scene of products such as cables, tablets, and wooden boards, but can also include other computer vision tasks, such as defect detection of road surfaces, chip surfaces, or any other object surfaces. In order to explain the application of the present invention by taking examples, only some features such as color and shape that readers can easily understand are listed. However, it does not mean that the present invention extracts and relies only on these features. Advantageously, embodiments of the present invention can be implemented based on Convolutional Neural Networks (CNNs) that can extract more complex features at multiple semantic levels, rather than extracting only limited low-level features.
[0017] 1 is a flow chart of a neural network training method 100 according to an embodiment of the present invention. The neural network training method according to the embodiment of the present invention will be described below with reference to FIG.
[0018] In step S101, a training image is input, and a perceptual map of the training image is calculated.
[0019] In this step, the training images input to train the neural network may be images of products or objects without defects, and the training images may be images of the same products or objects taken at different angles, different lighting conditions, and different backgrounds, so as to make the training samples as rich as possible, thereby further improving the defect detection accuracy of the trained neural network.
[0020] After inputting the training images, a noticeable map (also referred to as a noticeable map) of the training images may be calculated according to the input training images, and the calculated noticeable map of the training images may be a multi-scale perceptual map. Thus, the multi-scale perceptual map of the training images can represent the intensity change of each image feature in the training images, i.e., represent the visual salience difference of each image feature in the training images. The multi-scale perceptual map of the training images can make the generated training defect images more natural, and make the subsequent defect detection and localization process more accurate.
[0021] In one example, a multi-scale just-noticeable image of the training image can be calculated based on the input training image, e.g., just-noticeable difference (JND) can be calculated based on the input training image. The minimum perceptual map of the training images can be calculated by a just-in-time distortion (JND) model. Thus, the JND value is an important visual saliency index that describes the degree of human perception of image quality. The JND value reveals the perceptual threshold of image intensity changes that the human visual system can notice. Any perceptible distortion level must be greater than the JND value. In an embodiment of the present invention, a training defect image sample set with diversity and multi-scale samples can be generated by the JND model and used in the subsequent defect detection and location process.
[0022] Preferably, the process of calculating the multi-scale minimum perceptual map of the training image by the just noticeable difference model can include the following: First, calculate the minimum perceptual map of the training image based on the just noticeable difference model, and use a plurality of coefficients to process the minimum perceptual map of the training image (for example, multiply the plurality of coefficients with the minimum perceptual map of the training image respectively) to obtain the multi-scale minimum perceptual map of the training image. For example, select three coefficients (for example, 0.33, 0.66, and 1) to respectively multiply the minimum perceptual map of the training image, so as to obtain the three-scale minimum perceptual maps of the training image corresponding to three different scales. In another example, select four coefficients (for example, 0.25, 1, 0.75, and 1) to respectively multiply the minimum perceptual map of the training image, so as to obtain the four-scale minimum perceptual maps of the training image corresponding to four different scales. The above generation manner for the multi-scale minimum perceptual map of the training image is merely an example, and the embodiment of the present invention can generate the multi-scale minimum perceptual maps of the training pattern at various scales in various manners according to the needs of different application scenes. As described above, the multi-scale minimum perceptual map of the training image can represent the range and intensity variations of each image feature in the training image, and since it is perceptible to humans, the distribution and variations of the grayscale values therein represent the visual salience differences of each image feature in the training image.
[0023] In step S102, a training defect image is generated from at least the training image and the perceptual map of the training image, where the training defect image includes one or more defects generated based on the perceptual map of the training image.
[0024] In this step, the process of generating a training defect image can include first generating a defect mask image, and generating one or more defect texture images based on the training image. Then, based on a defect control panel and the defect mask image, and using a multi-scale perceptual map of the training image, the training image can be fused with the one or more defect texture images to generate a training defect image, where the defect control panel is used to control the proportion of the defect texture image that occupies the training image.
[0025] Preferably, the defect mask image can be generated by selecting one or more defect areas with different sizes and shapes from the training image. Also, preferably, the defect texture image can be generated randomly from any image, or can be generated by processing the training image. In one example, in order to enhance the realistic effect of simulating defects, the defect texture image can be generated by the result of enhancing the training image, for example, by performing various random processing including inversion, scaling, rotation, area cropping, color conversion, brightness change, exposure change, etc. on the training image, defect texture images with various textures can be obtained. FIG. 2 is an example of a defect mask image according to an embodiment of the present invention. Specifically, FIG. 2 shows a multi-scale defect mask image with markings of defect areas of different sizes, which is generated by selecting one or more defect areas with different sizes and shapes from the training image.
[0026] After obtaining the defect mask image and the defect texture image, preferably, according to the defect area defined by the combination of the defect control panel and the defect mask image, the training image and one or more defect texture images are fused using the multi-scale perceptual map of the training image as a weight image to generate a training defect image with various simulated defects. According to the corresponding principle of the perceptual map, the perceptual map of the training image can reveal the areas with high color contrast and saturation in the training image, and the areas with larger gray scale in the perceptual map can occupy more defect components in the training defect image, that is, can give more weight to the defect texture image. Conversely, the areas with smaller gray scale in the perceptual map of the training image can give more weight relatively to the input training image. On this basis, the application of the multi-scale perceptual map of the training image can obtain a training defect image fused at various scales, making the obtained training defect image more realistic, which can be further used in the subsequent defect detection process.
[0027] FIG. 3 shows various examples of defect control panels according to an embodiment of the present invention, and the defect control panel shown in FIG. 3 is used to limit the area occupied by the defect texture image when the defect texture image is fused with the training image. When used, the defect control panel to be used may be selected randomly from the multiple defect control panels, or the defect control panel to be used may be selected based on a preset order or criterion, and is not limited here. Preferably, the defect texture image may occupy an area of n% width and m% height at a certain position (e.g., upper, lower, left or right) in the training image, where m and n may be selected randomly from a preset interval, for example, the preset interval may be [50, 80]. In addition, the defect control panel may be, for example, an area covering the entire training image, and is not limited here. Of course, the above defect control panel area setting and size selection are merely examples, and in actual applications, any control method of the defect control panel may be selected based on the application scene, and the description is omitted here. By selecting and using the defect control panel, the proportion of the defect texture image that occupies the entire area of the training image can be controlled during the process of generating the training defect image, making the training defect image generation results more diverse and randomized, and obtaining more accurate neural network training and defect detection results.
[0028] Preferably, when the multi-scale perceptual map of the training image is generated, a multi-scale defect mask image can be generated accordingly, so as to fuse the training image and the defect texture image, and finally obtain a training defect image. Specifically, the multi-scale defect mask image can be a defect mask image marked with defect areas of different sizes corresponding to each scale in the multi-scale perceptual map of the training image. For example, referring to the above-mentioned process of generating the multi-scale minimally noticeable image perceptual map of the training image, the initially generated defect mask image can also be processed accordingly by multiple coefficients (for example, multiplying the defect mask image by multiple coefficients respectively) to obtain corresponding multi-scale defect mask images, so as to define the range of different defect areas at different scales. In the subsequent defect generation process, in the process of fusing the training image and the defect texture image by these multi-scale defect mask images based on the defect control panel, corresponding image processing can be performed on the defect areas marked on the multi-scale defect mask image, for example, a method such as Gaussian blur can be used to process each multi-scale defect area in the fusion process, so as to fuse and obtain a more realistic and natural training defect image.
[0029] The specific manner of generating the training defect image described above is merely an example. In actual applications, various manners can be used to generate the training defect image according to the corresponding scene, and is not limited thereto.
[0030] In step S103, reconstruction is performed on the training defect image by a reconstruction network including a first submodule, and a training reconstructed image with defects removed after reconstruction and a perceptual map of the training reconstructed image are obtained, where the first submodule is used to screen and process features by screening feature channels and extracting spatial relationships between each channel.
[0031] FIG. 4 is a diagram of the structure of a reconstruction network according to an embodiment of the present invention. In FIG. 4, a training defect image passes through a reconstruction network including an encoder and a decoder to generate a training reconstruction image with defects removed and a perceptual map (or a minimum perceptual map) of the training reconstruction image. As shown, a base sub-module and a first sub-module may be provided in each layer of the encoder and decoder of the reconstruction network, respectively, and the base sub-module and the first sub-module are connected to each other in each layer. Preferably, the base sub-module can include two successive 3×3 convolution layers, a normalization layer, and a reLU active layer to extract features of each layer. The first sub-module can include a channel attention sub-block and a spatial attention sub-block, and preferably, the channel attention sub-block can extract the relationship between different feature channels by an average pooling layer and two successive 1×1 convolution layers, and can screen relatively important feature channels by adding the relationship between different feature channels back to the training defect image. The spatial attention sub-block can further extract the spatial relationship of each feature channel, and the extracted spatial relationships can be cascaded together after passing through an average pooling layer and a max pooling layer, and then a standard convolution layer and a dilated convolution network (e.g., a 3×3 convolution layer with dilation) are used to screen important feature regions, where the presence of the dilated convolution network can expand the feature receptive field under the premise of maintaining spatial resolution.
[0032] When the input training defect image enters the reconstruction network, the encoder can be used to extract the input features first. Preferably, the convolution, normalization, and pooling processes of different parameters of the encoder in the reconstruction network can be used to extract features of different hierarchical levels. First, a convolution kernel can be used to perform convolution on the input training defect image to obtain a convolution mapping. Then, a linear correction unit and a batch normalization method can be used to normalize the convolution mapping to obtain a normalized convolution mapping. Then, a maximum or average pooling process can be applied to the normalized convolution mapping. In order to obtain rich multi-scale features, the encoder can adjust related parameters and repeat the above process multiple times to extract corresponding multi-scale feature maps through multiple downsampling processes. After the encoder process of the reconstruction network, the decoder can be used to restore the resolution of the multi-scale feature map through convolution, normalization, and upsampling processes. The reconstruction network combines the features of the encoder and the decoder with the same resolution and uses multiple sets of convolution processes. During the training phase of the neural network, the reconstruction network can convert the input training defect image into a corresponding training reconstructed image and a perceptual map of the corresponding training reconstructed image by learning and updating each parameter and weight of the neural network under supervision of the input training image and the perceptual map of the training image.
[0033] In step S104, the neural network is trained and parameters of the neural network are adjusted by performing defect detection based on the training images, the training reconstructed images, and a perceptual map of the training reconstructed images.
[0034] Preferably, performing defect detection based on the training image, the training reconstructed image, and the perceptual map of the training reconstructed image may include: extracting differences between the training image and the training reconstructed image, between the training image and the perceptual map of the training reconstructed image, and between the training image, the training reconstructed image, and the perceptual map of the training reconstructed image, respectively, by a division network including a second submodule to perform defect detection.
[0035] FIG. 5 shows a structural schematic diagram of a segmentation network according to an embodiment of the present invention. Preferably, the segmentation network in FIG. 5 can include a second sub-module and an encoder and a decoder connected to the second sub-module. In FIG. 5, the second sub-module is used to extract difference features between multi-layer training images, training reconstructed images, and the perceptual map of the training reconstructed images by inputting the training images, training reconstructed images, and the perceptual map of the training reconstructed images. First, the second sub-module first extracts the differences between the training images and the training reconstructed images, between the training images and the perceptual map of the training reconstructed images, and between the training images and the training reconstructed images and the perceptual map of the training reconstructed images, respectively, and then serially combines the differences to form a difference feature description whose number of feature channels is equal to the sum of the feature channels of these images, and feeds it into a network including an encoder and a decoder to perform defect segmentation, obtain the result of defect detection, and locate the defects. In FIG. 5, the specific network configuration of the encoder and the decoder is the same as that in FIG. 4, and the first sub-module may be installed in each layer of the encoder and the decoder, which will not be described again here.
[0036] The defect segmentation and detection of the segmentation network of FIG. 5 can locate defects in training images and provide a decision or probability indication of whether a defect is present.
[0037] Preferably, when locating the defect, the segmentation network can segment and classify the pixel values in the image for displaying the defect obtained, so that the location of the defect can be more intuitively shown. For example, when outputting the image for displaying the defect, the pixels in the grayscale image can be divided into three parts according to their grayscale values, and the normalized results can be represented as 0, 0.5, and 1, respectively, so that the location of the defect can be more clearly displayed. Needless to say, the above process of segmenting, classifying, and displaying the defect is only an example, and in actual application, various pixel value classification and normalization operations can be further performed on the image for displaying the defect according to needs, and are not limited here. After locating and displaying the defect, the segmentation network can output a judgment on whether the training image is a defect image or not as a result of defect detection, for example, whether the training image is a defect image or not.
[0038] The above-mentioned neural network training method of an embodiment of the present invention improves the training manner of the neural network and improves the accuracy of defect detection by introducing a first sub-module for screening and processing features through screening on feature channels and extracting the spatial relationship of each channel, or further introducing a second sub-module for extracting differences between the training image, the training reconstructed image, and the perceptual map of the training reconstructed image, thereby realizing accurate generation and robust detection of disordered defects for different types of products or objects, avoiding the process of collecting a large amount of defect data and manual labeling, and improving the user experience.
[0039] In addition, according to another embodiment of the present invention, it is not limited to performing neural network training by the specific operation process of the neural network training method, and other needs in various application scenarios can be realized through the neural network architecture and process. For example, the neural network architecture can realize the creation of diversified samples, such as human faces, animated images, etc., or can be applied to corresponding scenes that require the introduction of other image defects or noise, so as to introduce various relatively realistic simulated defects or noise into the image. The method according to the embodiment of the present invention can combine training images to introduce various noises to obtain diversified samples for application in various application scenarios.
[0040] Fig. 6 is a flow chart of a method 600 for detecting defects using a neural network according to an embodiment of the present invention. In this embodiment, defect detection can be performed using a neural network that has been trained according to the process shown in Fig. 1. Hereinafter, the defect detection method using a neural network according to the embodiment of the present invention will be described with reference to Fig. 6.
[0041] In step S601, a detected image is input, and a reconstruction network including a first submodule performs reconstruction on the detected image, and obtains a reconstructed image with defects removed after reconstruction and a perceptual map of the reconstructed image, where the first submodule is used to screen and process features by screening feature channels and extracting spatial relationships between each channel.
[0042] In this step, the detected image can be input to a reconstruction network including a first submodule in the neural network trained by the process shown in FIG. 1 to be reconstructed. Preferably, the detected image can be an image without defects or an image with one or more defects. In the process of detecting the detected image, a perceptual map of the reconstructed image can be obtained at the same time as obtaining a reconstructed image with the defects removed, or a multi-scale perceptual map of the reconstructed image can be further obtained for use in the subsequent defect detection and location process. The specific structure of the reconstruction network is shown in FIG. 4, and will not be described again here.
[0043] When the input detected image enters the reconstruction network, the encoder can be used to extract the input features first. Preferably, the convolution, normalization, and pooling processes of different parameters of the encoder in the reconstruction network can be used to extract features of different hierarchical levels. First, a convolution kernel can be used to perform convolution on the input detected image to obtain a convolution mapping. Then, a linear correction unit and a batch normalization method can be used to normalize the convolution mapping to obtain a normalized convolution mapping. Then, a maximum or average pooling process can be applied to the normalized convolution mapping. In order to obtain rich multi-scale features, the encoder can adjust related parameters and repeat the above process multiple times to extract corresponding multi-scale feature maps through multiple downsampling processes. After the encoder process of the reconstruction network, the decoder can be used to restore the resolution of the multi-scale feature maps through convolution, normalization, and upsampling processes. The reconstruction network combines the features of the encoder and decoder with the same resolution, and uses multiple sets of convolution processes to convert the input detected image into a corresponding reconstructed image with defects removed and a perceptual map of the corresponding reconstructed image.
[0044] In step S602, defect segmentation is performed based on the to-be-detected image, the reconstructed image, and a perceptual map of the reconstructed image to obtain a defect detection result.
[0045] Preferably, performing defect segmentation based on at least the detected image, the reconstructed image, and the perceptual map of the reconstructed image may include: A segmentation network including a second submodule extracts differences between the detected image and the reconstructed image, between the detected image and the perceptual map of the reconstructed image, and between the detected image and the reconstructed image and the perceptual map of the reconstructed image, respectively, to perform defect detection. The specific structure of the segmentation network is shown in FIG. 5, and will not be described again here.
[0046] The defect segmentation and detection of the segmentation network of FIG. 5 can locate defects in the detected image and provide a determination or probability indication of whether a defect is present.
[0047] Preferably, when locating the defect, the segmentation network can segment and classify the pixel values in the image for displaying the defect obtained, so that the location of the defect can be more intuitively shown. For example, when outputting the image for displaying the defect, the pixels in the grayscale image can be divided into three parts according to their grayscale values, and the normalized results can be represented as 0, 0.5, and 1, respectively, so that the location of the defect can be more clearly displayed. Needless to say, the above process of segmenting, classifying, and displaying the defect is only an example, and in actual application, various pixel value classification and normalization operations can be further performed on the image for displaying the defect according to needs, and are not limited here. After locating and displaying the defect, the segmentation network can output a judgment on whether the detected image is a defective image or not as a result of defect detection, for example, whether the detected image is a defective image or not.
[0048] The above-mentioned neural network defect detection method according to an embodiment of the present invention can improve the accuracy of defect detection by introducing a first sub-module for screening and processing features through screening on feature channels and extracting spatial relationships between each channel, or further introducing a second sub-module for extracting differences between the training image, the training reconstructed image, and the perceptual map of the training reconstructed image, thereby realizing accurate generation and robust detection of unregulated defects for different types of products or objects, avoiding the process of collecting large amounts of defect data and manually labeling, and improving user experience.
[0049] Next, a neural network training apparatus according to an embodiment of the present invention will be described with reference to FIG. 7. FIG. 7 is a block diagram of a neural network training apparatus 700 according to an embodiment of the present invention. As shown in FIG. 7, the neural network training apparatus 700 includes an input unit 710, a generation unit 720, a reconstruction unit 730, and a training unit 740. The neural network training apparatus 700 may include other components in addition to the above, but these components are not related to the content of the embodiment of the present invention, so they will not be illustrated or described here. In addition, the specific details of the following operations performed by the neural network training apparatus 700 according to the embodiment of the present invention are the same as the details described in FIG. 1 above, so that repeated descriptions of the same details will be omitted to avoid duplication.
[0050] An input unit 710 of the neural network training apparatus 700 of FIG. 7 inputs training images and calculates a perceptual map of the training images.
[0051] An input unit 710 inputs a training image and calculates a perceptible image of the training image.
[0052] The training images input by the input unit 710 for training the neural network may be images of products or objects without defects. In order to make the training samples as rich as possible, the training images may be images of the same product or object taken at different angles, different lighting conditions, and different backgrounds, so as to further improve the defect detection accuracy of the trained neural network.
[0053] After inputting the training image, a noticeable map of the training image may be calculated according to the input training image, and the calculated noticeable map of the training image may be a multi-scale perceptual map. Thus, the multi-scale perceptual map of the training image can represent the intensity change of each image feature in the training image, i.e., represent the visual salience difference of each image feature in the training image. The multi-scale perceptual map of the training image can make the generated training defect image more natural, and make the subsequent defect detection and localization process more accurate.
[0054] In one example, given a training image, a multi-scale just-noticeable image of the training image can be calculated, e.g., just-noticeable difference (JND) can be calculated. The minimum perceptual map of the training images can be calculated by a just-in-time distortion (JND) model. Thus, the JND value is an important visual saliency index that describes the degree of human perception of image quality. The JND value reveals the perceptual threshold of image intensity changes that the human visual system can notice. Any perceptible distortion level must be greater than the JND value. In an embodiment of the present invention, a training defect image sample set with diversity and multi-scale samples can be generated by the JND model and used in the subsequent defect detection and location process.
[0055] Preferably, the process of calculating the multi-scale minimum perceptual map of the training image by the just noticeable difference model can include the following: First, calculate the minimum perceptual map of the training image based on the just noticeable difference model, and use a plurality of coefficients to process the minimum perceptual map of the training image (for example, multiply the plurality of coefficients with the minimum perceptual map of the training image respectively) to obtain the multi-scale minimum perceptual map of the training image. For example, select three coefficients (for example, 0.33, 0.66, and 1) to respectively multiply the minimum perceptual map of the training image, so as to obtain the three-scale minimum perceptual maps of the training image corresponding to three different scales. In another example, select four coefficients (for example, 0.25, 1, 0.75, and 1) to respectively multiply the minimum perceptual map of the training image, so as to obtain the four-scale minimum perceptual maps of the training image corresponding to four different scales. The above generation manner for the multi-scale minimum perceptual map of the training image is merely an example, and the embodiment of the present invention can generate the multi-scale minimum perceptual maps of the training pattern at various scales in various manners according to the needs of different application scenes. As described above, the multi-scale minimum perceptual map of the training image can represent the range and intensity variations of each image feature in the training image, and since it is perceptible to humans, the distribution and variations of the grayscale values therein represent the visual salience differences of each image feature in the training image.
[0056] A generating unit 720 generates a training defect image from at least the training image and a perceptual map of the training image, the training defect image including one or more defects generated based on the perceptual map of the training image.
[0057] The process of generating a training defect image by the generating unit 720 may include first generating a defect mask image, and generating one or more defect texture images based on the training image. Then, based on a defect control panel and the defect mask image, and using a multi-scale perceptual map of the training image, the training image can be fused with the one or more defect texture images to generate a training defect image, where the defect control panel is used to control the proportion of the defect texture image that occupies the training image.
[0058] Preferably, the defect mask image can be generated by selecting one or more defect areas with different sizes and shapes from the training image. Also, preferably, the defect texture image can be generated randomly from any image, or can be generated by processing the training image. In one example, in order to enhance the realistic effect of simulating defects, the defect texture image can be generated by the result of enhancing the training image, for example, by performing various random processing including inversion, scaling, rotation, area cropping, color conversion, brightness change, exposure change, etc. on the training image, defect texture images with various textures can be obtained. FIG. 2 is an example of a defect mask image according to an embodiment of the present invention. Specifically, FIG. 2 shows a multi-scale defect mask image with markings of defect areas of different sizes, which is generated by selecting one or more defect areas with different sizes and shapes from the training image.
[0059] After obtaining the defect mask image and the defect texture image, preferably, according to the defect area defined by the combination of the defect control panel and the defect mask image, the training image and one or more defect texture images are fused using the multi-scale perceptual map of the training image as a weight image to generate a training defect image with various simulated defects. According to the corresponding principle of the perceptual map, the perceptual map of the training image can reveal the areas with high color contrast and saturation in the training image, and the areas with larger gray scale in the perceptual map can occupy more defect components in the training defect image, that is, can give more weight to the defect texture image. Conversely, the areas with smaller gray scale in the perceptual map of the training image can give more weight relatively to the input training image. On this basis, the application of the multi-scale perceptual map of the training image can obtain a training defect image fused at various scales, making the obtained training defect image more realistic, which can be further used in the subsequent defect detection process.
[0060] FIG. 3 shows various examples of defect control panels according to an embodiment of the present invention, and the defect control panel shown in FIG. 3 is used to limit the area occupied by the defect texture image when the defect texture image is fused with the training image. When used, the defect control panel to be used may be selected randomly from the multiple defect control panels, or the defect control panel to be used may be selected based on a preset order or criterion, and is not limited here. Preferably, the defect texture image may occupy an area of n% width and m% height at a certain position (e.g., upper, lower, left or right) in the training image, where m and n may be selected randomly from a preset interval, for example, the preset interval may be [50, 80]. In addition, the defect control panel may be, for example, an area covering the entire training image, and is not limited here. Of course, the above defect control panel area setting and size selection are merely examples, and in actual applications, any control method of the defect control panel may be selected based on the application scene, and the description is omitted here. By selecting and using the defect control panel, the proportion of the defect texture image that occupies the entire area of the training image can be controlled during the process of generating the training defect image, making the training defect image generation results more diverse and randomized, and obtaining more accurate neural network training and defect detection results.
[0061] Preferably, when the multi-scale perceptual map of the training image is generated, a multi-scale defect mask image can be generated accordingly, so as to fuse the training image and the defect texture image, and finally obtain a training defect image. Specifically, the multi-scale defect mask image can be a defect mask image marked with defect areas of different sizes corresponding to each scale in the multi-scale perceptual map of the training image. For example, referring to the above-mentioned process of generating the multi-scale minimum perceptual map of the training image, the initially generated defect mask image can also be processed accordingly by multiple coefficients (for example, multiplying the multiple coefficients with the defect mask image respectively) to obtain corresponding multi-scale defect mask images, so as to define the range of different defect areas at different scales. In the subsequent defect generation process, in the process of fusing the training image and the defect texture image by these multi-scale defect mask images based on the defect control panel, corresponding image processing can be performed on the defect areas marked on the multi-scale defect mask image, for example, a method such as Gaussian blur can be used to process each multi-scale defect area in the fusion process, so as to fuse and obtain a more realistic and natural training defect image.
[0062] The specific manner of generating the training defect image described above is merely an example. In actual applications, various manners can be used to generate the training defect image according to the corresponding scene, and is not limited thereto.
[0063] The reconstruction unit 730 performs reconstruction on the training defect image through a reconstruction network including a first submodule, and obtains a training reconstructed image with defects removed after reconstruction and a perceptual map of the training reconstructed image, where the first submodule is used for screening and processing features by screening feature channels and extracting spatial relationships between each channel.
[0064] FIG. 4 is a diagram of the structure of a reconstruction network according to an embodiment of the present invention. In FIG. 4, a training defect image passes through a reconstruction network including an encoder and a decoder to generate a training reconstruction image with defects removed and a perceptual map (or a minimum perceptual map) of the training reconstruction image. As shown, a base sub-module and a first sub-module may be provided in each layer of the encoder and decoder of the reconstruction network, respectively, and the base sub-module and the first sub-module are connected to each other in each layer. Preferably, the base sub-module can include two successive 3×3 convolution layers, a normalization layer, and a reLU active layer to extract features of each layer. The first sub-module can include a channel attention sub-block and a spatial attention sub-block, and preferably, the channel attention sub-block can extract the relationship between different feature channels by an average pooling layer and two successive 1×1 convolution layers, and can screen relatively important feature channels by adding the relationship between different feature channels back to the training defect image. The spatial attention sub-block can further extract the spatial relationships of each feature channel, and the extracted spatial relationships can be cascaded together after passing through an average pooling layer and a max pooling layer, and then a standard convolutional layer and a dilated convolutional network (e.g., a 3×3 convolutional layer with dilation) are used to screen important feature regions, where the presence of the dilated convolutional network can expand the feature receptive field under the premise of maintaining spatial resolution.
[0065] When the input training defect image enters the reconstruction network, the encoder can be used to extract the input features first. Preferably, the convolution, normalization, and pooling processes of different parameters of the encoder in the reconstruction network can be used to extract features of different hierarchical levels. First, a convolution kernel can be used to perform convolution on the input training defect image to obtain a convolution mapping. Then, a linear correction unit and a batch normalization method can be used to normalize the convolution mapping to obtain a normalized convolution mapping. Then, a maximum or average pooling process can be applied to the normalized convolution mapping. In order to obtain rich multi-scale features, the encoder can adjust related parameters and repeat the above process multiple times to extract corresponding multi-scale feature maps through multiple downsampling processes. After the encoder process of the reconstruction network, the decoder can be used to restore the resolution of the multi-scale feature map through convolution, normalization, and upsampling processes. The reconstruction network combines the features of the encoder and the decoder with the same resolution and uses multiple sets of convolution processes. During the training phase of the neural network, the reconstruction network can convert the input training defect image into a corresponding training reconstructed image and a perceptual map of the corresponding training reconstructed image by learning and updating each parameter and weight of the neural network under supervision of the input training image and the perceptual map of the training image.
[0066] A training unit 740 trains the neural network and adjusts parameters of the neural network by performing defect detection based on the training images, the training reconstructed images, and a perceptual map of the training reconstructed images.
[0067] Preferably, the training unit 740 performs defect detection by extracting differences between the training image and the training reconstructed image, between the training image and the perceptual map of the training reconstructed image, and between the training image, the training reconstructed image, and the perceptual map of the training reconstructed image, respectively, through a division network including a second sub-module.
[0068] FIG. 5 shows a structural schematic diagram of a segmentation network according to an embodiment of the present invention. Preferably, the segmentation network in FIG. 5 can include a second sub-module and an encoder and a decoder connected to the second sub-module. In FIG. 5, the second sub-module is used to extract difference features between multi-layer training images, training reconstructed images, and the perceptual map of the training reconstructed images by inputting the training images, training reconstructed images, and the perceptual map of the training reconstructed images. First, the second sub-module first extracts the differences between the training images and the training reconstructed images, between the training images and the perceptual map of the training reconstructed images, and between the training images and the training reconstructed images and the perceptual map of the training reconstructed images, respectively, and then serially combines the differences to form a difference feature description whose number of feature channels is equal to the sum of the feature channels of these images, and feeds it into a network including an encoder and a decoder to perform defect segmentation, obtain the result of defect detection, and locate the defects. In FIG. 5, the specific network configuration of the encoder and the decoder is the same as that in FIG. 4, and the first sub-module may be installed in each layer of the encoder and the decoder, which will not be described again here.
[0069] The defect segmentation and detection of the segmentation network of FIG. 5 can locate defects in training images and provide a decision or probability indication of whether a defect is present.
[0070] Preferably, when locating the defect, the segmentation network can segment and classify the pixel values in the image for displaying the defect obtained, so that the location of the defect can be more intuitively shown. For example, when outputting the image for displaying the defect, the pixels in the grayscale image can be divided into three parts according to their grayscale values, and the normalized results can be represented as 0, 0.5, and 1, respectively, so that the location of the defect can be more clearly displayed. Needless to say, the above process of segmenting, classifying, and displaying the defect is only an example, and in actual application, various pixel value classification and normalization operations can be further performed on the image for displaying the defect according to needs, and are not limited here. After locating and displaying the defect, the segmentation network can output a judgment on whether the training image is a defect image or not as a result of defect detection, for example, whether the training image is a defect image or not.
[0071] The above-mentioned neural network training apparatus of an embodiment of the present invention improves the training method of the neural network and improves the accuracy of defect detection by introducing a first sub-module for screening and processing features through screening on feature channels and extracting the spatial relationship of each channel, or further introducing a second sub-module for extracting differences between the training image, the training reconstructed image, and the perceptual map of the training reconstructed image, thereby realizing accurate generation and robust detection of disordered defects for different types of products or objects, avoiding the process of collecting a large amount of defect data and manually labeling, and improving the user experience.
[0072] Next, a neural network training apparatus according to an embodiment of the present invention will be described with reference to Fig. 8. Fig. 8 is a block diagram of a neural network training apparatus 800 according to an embodiment of the present invention. As shown in Fig. 8, the apparatus 800 can be a computer or a server.
[0073] As shown in Figure 8, neural network training apparatus 800 includes one or more processors 810 and memory 820, in addition to which neural network training apparatus 800 may include input devices, output devices (not shown), etc., which may be interconnected via a bus system and / or other types of connection mechanisms. It should be noted that the components and configuration of neural network training apparatus 800 shown in Figure 8 are exemplary only and not limiting, and neural network training apparatus 800 may have other components and configurations as needed.
[0074] The processor 810 may be a central processing unit (CPU) or other type of processing unit having data processing and / or command execution capabilities, and may use computer program commands stored in the memory 820 to perform desired functions, including: inputting a training image and calculating a perceptual map of the training image; generating a training defect image from at least the training image and the perceptual map of the training image, where the training defect image includes one or more defects generated based on the perceptual map of the training image; performing reconstruction on the training defect image through a reconstruction network including a first sub-module, and obtaining a training reconstructed image and a perceptual map of the training reconstructed image with defects removed after reconstruction, where the first sub-module is used for screening and processing features by screening on feature channels and extracting spatial relationships of each channel; performing defect detection based on the training image, the training reconstructed image, and the perceptual map of the training reconstructed image to train the neural network and adjust parameters of the neural network.
[0075] The memory 820 may include one or more computer program products, which may include various types of computer-readable recording media, such as, for example, volatile and / or non-volatile memory. One or more computer program commands may be stored in the computer-readable recording media, and the processor 810 may execute the program commands to perform the functions of the neural network training device according to the embodiment of the present invention and / or other desired functions and / or the neural network training method according to the embodiment of the present invention. Various application programs and various data may also be stored in the computer-readable recording media.
[0076] Hereinafter, a computer-readable storage medium storing a computer program according to an embodiment of the present invention will be described, and the computer program can be implemented by a processor to perform the following steps: input a training image and calculate a perceptual map of the training image; generate a training defect image from at least the training image and the perceptual map of the training image, where the training defect image includes one or more defects generated based on the perceptual map of the training image; perform reconstruction on the training defect image through a reconstruction network including a first sub-module, and obtain a training reconstructed image and a perceptual map of the training reconstructed image from which defects have been removed after reconstruction, where the first sub-module is used for screening and processing features by screening on feature channels and extracting spatial relationships of each channel; perform defect detection based on the training image, the training reconstructed image, and the perceptual map of the training reconstructed image to train the neural network and adjust parameters of the neural network.
[0077] Hereinafter, a defect detection apparatus using a neural network according to an embodiment of the present invention will be described with reference to FIG. 9. FIG. 9 shows a block diagram of an apparatus 900 for performing defect detection using a neural network according to an embodiment of the present invention. As shown in FIG. 9, the defect detection apparatus 900 using a neural network includes an input unit 910 and a detection unit 920. The apparatus 900 may include other components in addition to the above, but these components are not related to the content of the embodiment of the present invention, so they will not be illustrated or described here. In addition, the specific details of the following operations performed by the apparatus 900 according to the embodiment of the present invention are the same as the details described in FIG. 6 above, so to avoid duplication, the same details will not be described again.
[0078] The input unit 910 of the apparatus 900 for detecting defects using neural network in FIG. 9 inputs a detected image, and performs reconstruction on the detected image through a reconstruction network including a first submodule, to obtain a reconstructed image with defects removed after reconstruction and a perceptual map of the reconstructed image, in which the first submodule is used for screening and processing features through screening feature channels and extracting spatial relationships between each channel.
[0079] The input unit 910 can input the detected image to a reconstruction network including a first submodule in the neural network trained by the process shown in FIG. 1 for reconstruction. Preferably, the detected image can be an image without defects or an image with one or more defects. In the process of detecting the detected image, a perceptual map of the reconstructed image can be obtained at the same time as obtaining a reconstructed image with the defects removed, or a multi-scale perceptual map of the reconstructed image can be further obtained for use in the subsequent defect detection and location process. The specific structure of the reconstruction network is shown in FIG. 4, and will not be described again here.
[0080] When the input detected image enters the reconstruction network, the encoder can be used to extract the input features first. Preferably, the convolution, normalization, and pooling processes of different parameters of the encoder in the reconstruction network can be used to extract features of different hierarchical levels. First, a convolution kernel can be used to perform convolution on the input detected image to obtain a convolution mapping. Then, a linear correction unit and a batch normalization method can be used to normalize the convolution mapping to obtain a normalized convolution mapping. Then, a maximum or average pooling process can be applied to the normalized convolution mapping. In order to obtain rich multi-scale features, the encoder can adjust related parameters and repeat the above process multiple times to extract corresponding multi-scale feature maps through multiple downsampling processes. After the encoder process of the reconstruction network, the decoder can be used to restore the resolution of the multi-scale feature maps through convolution, normalization, and upsampling processes. The reconstruction network combines the features of the encoder and decoder with the same resolution, and uses multiple sets of convolution processes to convert the input detected image into a corresponding reconstructed image with defects removed and a perceptual map of the corresponding reconstructed image.
[0081] A detection unit 920 performs defect segmentation based on the detected image, the reconstructed image, and the perceptual map of the reconstructed image to obtain a defect detection result.
[0082] Preferably, the detection unit 920 uses a division network including a second sub-module to respectively extract the differences between the detected image and the reconstructed image, between the detected image and the perceptual map of the reconstructed image, and between the detected image and the reconstructed image and the perceptual map of the reconstructed image to perform defect detection. The specific structure of the division network is shown in FIG. 5, and will not be described again here.
[0083] The defect segmentation and detection of the segmentation network of FIG. 5 can locate defects in the detected image and provide a determination or probability indication of whether a defect is present.
[0084] Preferably, when locating the defect, the segmentation network can segment and classify the pixel values in the image for displaying the defect obtained, so that the location of the defect can be more intuitively shown. For example, when outputting the image for displaying the defect, the pixels in the grayscale image can be divided into three parts according to their grayscale values, and the normalized results can be represented as 0, 0.5, and 1, respectively, so that the location of the defect can be more clearly displayed. Needless to say, the above process of segmenting, classifying, and displaying the defect is only an example, and in actual application, various pixel value classification and normalization operations can be further performed on the image for displaying the defect according to needs, and are not limited here. After locating and displaying the defect, the segmentation network can output a judgment on whether the detected image is a defective image or not as a result of defect detection, for example, whether the detected image is a defective image or not.
[0085] The above-mentioned neural network defect detection apparatus of an embodiment of the present invention can improve the accuracy of defect detection by introducing a first sub-module for screening and processing features through screening on feature channels and extracting spatial relationships between each channel, or further introducing a second sub-module for extracting differences between the training image, the training reconstructed image, and the perceptual map of the training reconstructed image, thereby realizing accurate generation and robust detection of random defects for different types of products or objects, avoiding the process of collecting large amounts of defect data and manual labeling, and improving user experience.
[0086] Hereinafter, a defect detection apparatus using a neural network according to an embodiment of the present invention will be described with reference to Fig. 10. Fig. 10 shows a block diagram of an apparatus 900 for performing defect detection using a neural network according to an embodiment of the present invention. As shown in Fig. 10, the apparatus 1000 can be a computer or a server.
[0087] As shown in Figure 10, the device 1000 includes one or more processors 1010 and memory 1020, in addition to which the device 1000 may include input devices, output devices (not shown), etc., which may be interconnected via a bus system and / or other type of connection mechanism. It should be noted that the components and structure of the device 1000 shown in Figure 9 are exemplary only and not limiting. The device 1000 may have the components and structure as desired.
[0088] The processor 1010 may be a central processing unit (CPU) or other types of processing units having data processing and / or command execution capabilities, and may use computer program commands stored in the memory 1020 to perform desired functions, including: inputting a detected image, performing reconstruction on the detected image through a reconstruction network including a first sub-module, obtaining a reconstructed image with defects removed after reconstruction and a perceptual map of the reconstructed image, where the first sub-module is used for screening and processing features through screening feature channels and extracting spatial relationships between each channel; performing defect segmentation based on the detected image, the reconstructed image, and the perceptual map of the reconstructed image to obtain a defect detection result;
[0089] The memory 1020 may include one or more computer program products, which may include various types of computer readable storage media, such as, for example, volatile and / or non-volatile memory. One or more computer program commands may be stored in the computer readable storage media, and the processor 1010 may execute the program commands to perform the functions of the apparatus for defect detection by neural network training according to the embodiments of the present invention and / or other desired functions and / or the method for defect detection by neural network training according to the embodiments of the present invention. Various application programs and various data may also be stored in the computer readable storage media.
[0090] Hereinafter, a computer-readable storage medium storing a computer program according to an embodiment of the present invention will be described, and the computer program can be implemented by a processor to perform the following steps: input a detected image, perform reconstruction on the detected image by a reconstruction network including a first sub-module, obtain a reconstructed image with defects removed after reconstruction and a perceptual map of the reconstructed image, the first sub-module is used for screening and processing features by screening feature channels and extracting spatial relationships of each channel, and perform defect segmentation based on the detected image, the reconstructed image, and the perceptual map of the reconstructed image to obtain a defect detection result.
[0091] Of course, the above specific embodiments are merely examples and are not limiting, and those skilled in the art may combine and integrate some steps and devices from the above described embodiments based on the concept of the present invention to achieve the effects of the present invention. Such combined and integrated embodiments are also included in the present invention and will not be described here.
[0092] The advantages, advantages, effects, etc. mentioned in the present invention are only illustrative and are not limiting, and these advantages, advantages, effects are not essential to each embodiment of the present invention. In addition, the specific details disclosed in the above are only for the purpose of easy understanding of the action of the examples, and are not limiting. The above details do not limit the following. In other words, it is essential to use the above specific details to realize the present invention.
[0093] Block diagrams of components, devices, equipment, and systems according to the present invention are exemplary only, and do not require or suggest that the components, devices, equipment, and systems be necessarily connected, laid out, or arranged in the manner shown in the block diagrams. Those skilled in the art will recognize that the components, devices, equipment, and systems can be connected, laid out, or arranged in any manner. The words "including," "including," "having," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably. As used herein, the words "or" and "and" mean the words "and / or," and can be used interchangeably, unless the context clearly indicates otherwise. As used herein, the word "for example" means "including but not limited to," and can be used interchangeably.
[0094] The step flow charts and above description of the present invention are exemplary only and are not intended to require or suggest that the steps of each embodiment be performed in the order presented. As will be appreciated by those skilled in the art, the steps within the above embodiments may be performed in any order. Words such as "then," "thereafter," "next," and the like are not intended to limit the order of the steps. These words are only intended to guide the reader in reading these method descriptions. In addition, any reference to a singular element using the articles "a," "an," or "the" does not limit the element to the singular.
[0095] Furthermore, the steps and apparatus in each embodiment of this specification are not limited to being performed in a certain embodiment, and in fact, new embodiments can be constructed by combining some of the relevant steps and apparatus in each embodiment of this specification based on the concept of the present invention, and these new embodiments are also included in the scope of the present invention.
[0096] Each operation in the methods described above may be implemented by any suitable means capable of performing the corresponding function, including various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs), or processors.
[0097] Each illustrative logic block, module, circuit, etc. may be implemented or described using a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof, designed to perform the functions described herein. A general purpose processor may be a microprocessor, but alternatively, the processor may be any commercially available processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
[0098] The steps embodying the method or algorithm described herein may be embodied directly in hardware, in a software module executed by a processor, or a combination of the two. The software module may be stored on any form of tangible storage medium. Possible storage media include, for example, a random access memory (RAM), a read only memory (ROM), a fast flash memory, an EPROM memory, an EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, and so on. The storage medium is coupled to the processor such that the processor reads information from the storage medium, and writes information to the storage medium. In the alternative, the storage medium is integral to the processor. The software module may be a single command or multiple commands, and may be distributed over several different code segments, among different programs, and across multiple storage media.
[0099] The methods invented herein include one or more operations for achieving the described method. The methods and / or operations may be interchanged with one another without departing from the scope of the claims. In other words, except where a specific order of operations is specified, the order and / or execution of specific operations may be changed without departing from the scope of the claims.
[0100] The functions may be implemented by hardware, software, firmware, or any combination thereof. If implemented by software, the functions may be stored as one or more commands on a computer readable medium. The recording medium may be any available medium that can be accessed by a computer. The following examples are not limiting. Such computer readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic memory, or other determinable media that can carry or store desired program code in the form of commands or data structures and that can be accessed by a computer. As used herein, disc includes compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), soft magnetic discs, and Blu-ray discs.
[0101] Thus, a computer program product can perform the operations described herein. For example, such a computer program product can be a computer-readable tangible medium having instructions tangibly stored (and / or encoded) thereon, which instructions are executed by one or more processors to perform the operations described herein. The computer program product may include packaging materials.
[0102] The software or commands may be transmitted over a transmission medium, such as coaxial cable, fiber optics, twisted pair cable, Digital Subscriber Line (DSL), or wireless technologies such as infrared, radio or microwave, over which the software may be transmitted from a website, server, or other remote source.
[0103] Alternatively, modules and / or other suitable means for performing the methods and techniques described herein may be downloaded and / or otherwise obtained by a user terminal and / or base station at any suitable time. For example, such a device may be coupled to a server to facilitate the transmission of means for performing the methods described herein. Alternatively, the methods described herein may be provided on a storage medium (e.g., RAM, ROM, physical storage medium such as a CD or soft magnetic disk) such that a user terminal and / or base station obtains the methods when coupled to the device or provides the storage medium to the device. Alternatively, any other suitable technology for providing the methods and techniques described herein to a device may be used.
[0104] Other examples and implementations are within the scope and spirit of the present invention and the claims. For example, depending on the nature of the software, the functions described above may be implemented using software executed by a processor, hardware, firmware, hardwired, or any combination thereof. Features implementing the functions may also be physically located in various locations, including distribution such that portions of the functions are implemented in different physical locations. Moreover, as used herein, including as used in the claims, "or" in enumerations such as "at least one" means a disjunctive enumeration. That is, an enumeration such as "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Moreover, the term "exemplary" does not imply that the described example is optimal or better than other examples.
[0105] Various changes, substitutions, and modifications may be made to the techniques described herein without departing from the teachings of the claims. Moreover, the claims are not limited to the specific details of the processes, apparatus, manufacture, compositions of events, means, methods, and acts described herein. Any existing or developed processes, apparatus, manufacture, compositions of events, means, methods, or acts may be used to perform the same function or achieve substantially the same result as described herein. Accordingly, the claims include within their scope such processes, apparatus, manufacture, compositions of events, means, methods, or acts.
[0106] The above invention is provided to enable one skilled in the art to make or use the invention. Various modifications of these schemes will be apparent to those skilled in the art, and the general principles defined herein may be applied in other ways without departing from the scope of the invention. Thus, it is not desired that the present invention be limited to the schemes shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
[0107] The foregoing description has been presented for purposes of illustration and description, and is not intended to limit the embodiments of the present invention to the precise form disclosed herein. Having considered the foregoing exemplary embodiments and examples, certain variations, modifications, alterations, additions, and subcombinations thereof will be apparent to those skilled in the art.
Claims
1. A neural network training method executed by a neural network training device, comprising: inputting training images and computing a perceptual map of said training images; generating a training defect image based on at least the training image and a perceptual map of the training image, the training defect image including one or more defects generated based on the perceptual map of the training image; performing reconstruction on the training defect image by a reconstruction network including a first submodule to obtain a training reconstructed image with the defect removed after reconstruction and a perceptual map of the training reconstructed image, the first submodule being used for screening and processing features by screening on feature channels and extracting spatial relationships of each channel; and performing defect detection based on the training images, the training reconstructed images, and a perceptual map of the training reconstructed images to train a neural network and adjust parameters of the neural network.
2. The step of computing a perceptual map of the training images comprises:
2. The method of claim 1, further comprising computing a multi-scale perceptual map of the training images.
3. The step of generating training defect images based on at least the training images and the perceptual map of the training images comprises: generating a defect mask image and generating one or more defect texture images based on the training images; 2. The method of claim 1, further comprising: fusing the training image and the one or more defect texture images to generate a training defect image based on a defect control panel and the defect mask image and utilizing a multi-scale perceptual map of the training image, wherein the defect control panel is used to control the proportion of the defect texture image in the training image.
4. 2. The method of claim 1, wherein the first submodule includes a dilated convolutional network for expanding feature receptive fields.
5. 2. The method for training a neural network according to claim 1, wherein the reconstruction network comprises an encoder and a decoder, and a first sub-module is provided at each layer of the encoder and the decoder, respectively.
6. The step of performing defect detection based on at least the training images, the training reconstructed images, and a perceptual map of the training reconstructed images includes:
2. The method for training a neural network according to claim 1, further comprising: extracting, by a partitioning network including a second submodule, a difference between the training image and the training reconstructed image, a difference between the training image and the perceptual map of the training reconstructed image, and a difference between the training image and the training reconstructed image and the perceptual map of the training reconstructed image, respectively, to perform defect detection.
7. 7. The method for training a neural network according to claim 6, wherein the partitioning network comprises an encoder and a decoder, and a first sub-module is provided at each layer of the encoder and the decoder, respectively.
8. A defect detection method performed by a defect detection device, comprising: A step of inputting a detected image, performing reconstruction on the detected image by a reconstruction network including a first submodule, and obtaining a reconstructed image with defects removed after reconstruction and a perceptual map of the reconstructed image, wherein the first submodule is used for screening and processing features by screening feature channels and extracting spatial relationships between each channel; and performing defect segmentation based on the detected image, the reconstructed image, and a perceptual map of the reconstructed image to obtain a defect detection result.
9. an input unit for inputting training images and computing a perceptual map for said training images; a generation unit for generating a training defect image based on at least the training image and a perceptual map of the training image, the training defect image including one or more defects generated based on the perceptual map of the training image; a reconstruction unit for performing reconstruction on the training defect image by a reconstruction network including a first submodule, to obtain a training reconstructed image with defects removed after reconstruction and a perceptual map of the training reconstructed image, the first submodule being used for screening and processing features by screening on feature channels and extracting spatial relationships of each channel; and a training unit that performs defect detection based on the training image, the training reconstructed image, and a perceptual map of the training reconstructed image to train a neural network and adjust parameters of the neural network.
10. A processor; a memory in which computer program instructions are stored; The computer program instructions, when executed by the processor, cause the processor to: inputting training images and computing a perceptual map of said training images; generating a training defect image based on at least the training image and a perceptual map of the training image, the training defect image including one or more defects generated based on the perceptual map of the training image; performing reconstruction on the training defect image by a reconstruction network including a first submodule to obtain a training reconstructed image with the defect removed after reconstruction and a perceptual map of the training reconstructed image, the first submodule being used for screening and processing features by screening on feature channels and extracting spatial relationships of each channel; and performing defect detection based on the training images, the training reconstructed images, and a perceptual map of the training reconstructed images to train a neural network and adjust parameters of the neural network.
11. A computer program, which when executed by a processor: inputting training images and computing a perceptual map of said training images; generating a training defect image based on at least the training image and a perceptual map of the training image, the training defect image including one or more defects generated based on the perceptual map of the training image; performing reconstruction on the training defect image by a reconstruction network including a first submodule to obtain a training reconstructed image with the defect removed after reconstruction and a perceptual map of the training reconstructed image, the first submodule being used for screening and processing features by screening on feature channels and extracting spatial relationships of each channel; performing defect detection based on the training images, the training reconstructed images, and a perceptual map of the training reconstructed images to train a neural network and adjust parameters of the neural network.
