Defect detection method, device, equipment and storage medium based on image segmentation

Abstract

The present invention discloses a defect detection method based on image segmentation, the method comprising: receiving a training image containing a target defect, annotating the target defect with pixels, and obtaining a defect image; extracting features from the defect image through a preset neural network to obtain a target feature layer, and performing average pooling of the target feature layer at different scales to obtain pooling results at different scales; performing feature fusion on the pooling results at different scales to obtain a target feature map, and calculating the consistency loss based on the target feature map; iteratively processing the preset neural network based on the consistency loss, and adjusting the parameters of the preset neural network after the iterative processing to complete the training of the preset neural network. The present invention also discloses a defect detection device, equipment, and storage medium based on image segmentation. The present invention improves the defect compatibility problem in detection, increases the integrity of defects, reduces defect classification errors, and improves the stability and generalization of neural networks.

Description

Technical Field

[0001] The present invention relates to the field of image segmentation, and in particular to a defect detection method, device, equipment and storage medium based on image segmentation. Background Art

[0002] With the rapid development of deep learning technology, image segmentation technology has been applied to many fields such as medical treatment, transportation, and industry. In particular, the defect detection branch has been widely used in cloth defect detection, workpiece surface quality inspection, aerospace field, etc. In industrial scenarios, the types, shapes, and sizes of defects are very random. When solving defect detection problems, the current mainstream image segmentation network is often limited to the lack of flexibility in defect detection scale. It is difficult to be compatible with the detection accuracy of large targets and not miss the detection effect of small targets in the same scenario. In defect classification, due to differences in defect scales, misjudgment of voids, discontinuities, and edge categories in large-scale defect detection is also a thorny problem currently faced by defect detection. Summary of the invention

[0003] The main purpose of the present invention is to provide a defect detection method, device, equipment and storage medium based on image segmentation, aiming to solve the technical problem of low detection accuracy of existing image segmentation technology when performing defect detection.

[0004] In addition, to achieve the above object, the present invention also provides a defect detection method based on image segmentation, the defect detection method based on image segmentation comprises the following steps:

[0005] Receiving a training image containing a target defect, and annotating pixels of the target defect to obtain a defect image;

[0006] By using a preset neural network, feature extraction is performed on the defect image to obtain a target feature layer, and average pooling of different scales is performed on the target feature layer to obtain pooling results of different scales;

[0007] Performing feature fusion on the pooling results of different scales to obtain a target feature map, and calculating consistency loss based on the target feature map;

[0008] According to the consistency loss, the preset neural network is iteratively processed, and the parameters of the preset neural network after the iterative processing are adjusted to complete the training of the preset neural network.

[0009] Optionally, after the step of receiving a training image containing a target defect, annotating pixels of the target defect, and obtaining a defect image, the method further comprises:

[0010] Performing data enhancement processing on the defect image to obtain a transformed image, and establishing a consistent relationship between the defect image and the transformed image;

[0011] Determining whether the defect image and the transformed image meet the input requirements of a preset neural network;

[0012] If the defect image and the transformed image do not meet the input requirements of the preset neural network, normalizing and interpolating the defect image and the transformed image to obtain a target image;

[0013] The step of extracting features from the defect image by using a preset neural network to obtain a target feature layer comprises:

[0014] The target image is subjected to feature extraction via a preset neural network to obtain a target feature layer.

[0015] Optionally, after the step of determining whether the defect image and the transformed image meet the input requirements of a preset neural network, the following steps are included:

[0016] If the defect image and the transformed image meet the input requirements of the preset neural network, use the defect image and the transformed image as the first image;

[0017] The step of extracting features from the defect image by using a preset neural network to obtain a target feature layer comprises:

[0018] The first image is subjected to feature extraction through a preset neural network to obtain a target feature layer.

[0019] Optionally, the step of extracting features from the defect image by using a preset neural network to obtain a target feature layer, and performing average pooling of different scales on the target feature layer to obtain pooling results of different scales includes:

[0020] Acquire a target scale of the target feature layer and a scale set corresponding to the target scale, wherein the scale set includes at least two different scales;

[0021] According to different scales in the scale set, average pooling is performed on the target feature layer to obtain pooling results of different scales.

[0022] Optionally, the step of performing feature fusion on the pooling results of different scales to obtain a target feature map, and calculating the consistency loss according to the target feature map includes:

[0023] A target difference between the target feature map and the target feature layer is calculated, and the target difference is used as the consistency loss.

[0024] Optionally, the step of iteratively processing the preset neural network according to the consistency loss and adjusting the parameters of the iteratively processed preset neural network to complete the training of the preset neural network includes:

[0025] Obtaining a target convolution kernel used when performing average pooling of different scales on the target feature layer, and using the target convolution kernel as a parameter of the preset neural network;

[0026] Adjusting the target convolution kernel, and returning to the step of performing average pooling of different scales on the target feature layer to obtain pooling results of different scales, and obtaining a target consistency loss;

[0027] If the target consistency loss is greater than or equal to the preset threshold, return to the step of adjusting the target convolution kernel until the target consistency loss is less than the preset threshold, thereby completing the training of the preset neural network.

[0028] Optionally, after the step of iteratively processing the preset neural network according to the consistency loss and adjusting the parameters of the iteratively processed preset neural network to complete the training of the preset neural network, the method further comprises:

[0029] Inputting a detection image containing a first defect into the trained preset neural network;

[0030] Obtaining a probability map of the output of the trained preset neural network;

[0031] A defect type of the first defect is determined according to the probability map.

[0032] In addition, to achieve the above-mentioned purpose, the present invention also provides a defect detection device based on image segmentation, and the defect detection device based on image segmentation comprises:

[0033] A pixel annotation module is used to receive a training image containing a target defect, and perform pixel annotation on the target defect to obtain a defect image;

[0034] A feature extraction and pooling module is used to extract features from the defect image through a preset neural network to obtain a target feature layer, and perform average pooling of different scales on the target feature layer to obtain pooling results of different scales;

[0035] A consistency loss calculation module, used to perform feature fusion on the pooling results of different scales to obtain a target feature map, and calculate the consistency loss based on the target feature map;

[0036] The neural network training module is used to iteratively process the preset neural network according to the consistency loss, and adjust the parameters of the preset neural network after the iterative processing to complete the training of the preset neural network.

[0037] In addition, to achieve the above-mentioned purpose, the present invention also provides an image segmentation-based defect detection device, which includes: a memory, a processor, and an image segmentation-based defect detection program stored in the memory and executable on the processor, and when the program is executed by the processor, the steps of the image segmentation-based defect detection method as described above are implemented.

[0038] In addition, to achieve the above-mentioned purpose, the present invention also provides a storage medium, on which is stored an image segmentation-based defect detection program, and when the image segmentation-based defect detection program is executed by a processor, the steps of the image segmentation-based defect detection method as described above are implemented.

[0039] A defect detection method, device, equipment and storage medium based on image segmentation are proposed in an embodiment of the present invention. In the embodiment of the present invention, the defect detection program based on image segmentation receives a training image containing a target defect, and annotates the target defect with pixels to obtain a defect image, and then extracts features from the defect image through a preset neural network to obtain a target feature layer, and then averages the target feature layer at different scales to obtain pooling results at different scales, and then performs feature fusion on the pooling results at different scales to obtain a target feature map, and calculates the consistency loss based on the target feature map, and finally completes the training of the preset neural network by iteratively processing the preset neural network and adjusting the parameters of the preset neural network after iterative processing. By training the preset neural network, richer feature representations can be obtained during the neural network training process, and the consistent expression of defects is enhanced through consistency loss, thereby improving the defect compatibility problem that occurs during the detection process, increasing the integrity of defects, reducing defect classification errors, and thus improving the stability and generalization of the neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of the hardware structure of an implementation of an image segmentation-based defect detection device provided in an embodiment of the present invention;

[0041] Figure 2 It is a flowchart of a first embodiment of a defect detection method based on image segmentation according to the present invention;

[0042] Figure 3 It is a flow chart of a second embodiment of a defect detection method based on image segmentation according to the present invention;

[0043] Figure 4It is a schematic diagram of functional modules of an embodiment of a defect detection device based on image segmentation of the present invention.

[0044] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0045] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0046] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present invention, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0047] The defect detection terminal (also called terminal, device or terminal device) based on image segmentation in the embodiment of the present invention can be a PC, or a mobile terminal device with a display function such as a smart phone, a tablet computer and a portable computer.

[0048] like Figure 1 As shown, the terminal may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0049] Optionally, the terminal may also include a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, and the like. Among them, sensors include light sensors, motion sensors, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display screen according to the brightness of the ambient light, and the proximity sensor may turn off the display screen and / or backlight when the mobile terminal is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile terminal (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; of course, the mobile terminal may also be equipped with other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., which will not be repeated here.

[0050] Those skilled in the art will understand that Figure 1 The terminal structure shown in the figure does not constitute a limitation on the terminal, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0051] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a defect detection program based on image segmentation.

[0052] exist Figure 1 In the terminal shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the client (user end) and communicate data with the client; and the processor 1001 can be used to call the image segmentation-based defect detection program stored in the memory 1005. When the image segmentation-based defect detection program is executed by the processor, the operations in the image segmentation-based defect detection method provided in the following embodiments are implemented.

[0053] Based on the above-mentioned device hardware structure, an embodiment of the defect detection method based on image segmentation of the present invention is proposed.

[0054] Reference Figure 2 In a first embodiment of the defect detection method based on image segmentation of the present invention, the defect detection method based on image segmentation includes:

[0055] Step S10, receiving a training image containing a target defect, and annotating pixels of the target defect to obtain a defect image.

[0056] In this embodiment, the image segmentation-based defect detection method is applied to an image segmentation-based defect detection device, wherein the image segmentation-based defect detection device may be a personal computer.

[0057] The target defect in this embodiment refers to the defect of the product. Specifically, the application scenario of the defect detection method based on image segmentation in this embodiment is product defect detection. By taking pictures of defective products to obtain training images containing target defects, it can be known that pictures containing the same product defect are classified into one category, and the target defects in the training images are annotated at the pixel level. It is known that the pixel-level annotation of the training images is to determine the objects in the training images, and according to the process of name labeling, the pixel-level annotation can be segmented based on pixel color, that is, the target defect is different from the pixel color in the surrounding image. Pixel-level annotation of the target defect can make the selection of the object (target defect) more convenient and quick. Pixel annotation is also the most important work before image training.

[0058] Step S20, extracting features from the defect image through a preset neural network to obtain a target feature layer, and performing average pooling of different scales on the target feature layer to obtain pooling results of different scales.

[0059] It can be seen that the defect image is input into a preset neural network, and features of the defect image are extracted through the preset neural network. Specifically, since color is easily affected by light, before feature extraction of the defect image, the defect image can be gray-processed to speed up the feature extraction. It can be understood that there are many methods of feature extraction, such as scale-invariant feature transformation. Feature extraction is to scan the image to obtain the feature points shown in the image (prominent points that will not change due to external factors), such as corner points, edge points, and points with different intensities from the surrounding images. After finding the feature points, points with edge effects can also be removed to retain key points (for example, corner points when the image is rotated). The benefit of removing the points with edge effects is to enhance the anti-noise ability and stability. Then, these feature points are represented by numerical values, and features of different scales are extracted for the defective image to obtain multi-layer feature layers, each of which has a different feature matrix. This embodiment takes a feature layer of an intermediate scale, that is, the target feature layer in this embodiment, and performs average pooling of different scales on the target feature layer to obtain pooling results of different scales. Specifically, different scales can select the scale of the target feature layer, half the scale of the target feature layer, and one-third the scale of the target feature layer. The scale of the convolution kernel used for average pooling of different scales is large. Small can also be the scale of the target feature layer, half the scale of the target feature layer, and one-third the scale of the target feature layer, etc. After average pooling of the target feature layer at different scales, pooling results of different scales are obtained. The average pooling here refers to the average value of the eigenvalues ​​within the size of the convolution kernel used for average pooling. The pooling result obtained is actually a matrix. For example, the target feature layer is a 50×50 feature matrix, and the size of the convolution kernel used for average pooling is half of the target feature layer, that is, a 25×25 grid. The result of average pooling the target feature layer using the convolution kernel is a 2×2 matrix.

[0060] Step S30, performing feature fusion on the pooling results of different scales to obtain a target feature map, and calculating the consistency loss based on the target feature map.

[0061] It is understandable that fusing features of different scales is a method to improve segmentation performance. It can be seen that low-level features have higher resolution and contain more position and detail information, but have lower semantics and more noise, while high-level features have higher semantic information, but low resolution and poor perception of details. Feature fusion includes early fusion and late fusion. After feature fusion of pooling results of different scales, higher-dimensional features or composite features may be obtained. By optimizing the correlation between corresponding features between different pooling results, the target feature map is obtained. It can be seen that by calculating the consistency loss between the target feature map and the defect image, the difference in features of different defect images corresponding to the same product defect is obtained.

[0062] Step S40, iteratively processing the preset neural network according to the consistency loss, and adjusting the parameters of the preset neural network after the iterative processing to complete the training of the preset neural network.

[0063] After feature fusion of pooling results of different scales, consistency loss is obtained. The consistency loss here represents the difference in identifying defects of the same type and features of different scales. The smaller the consistency loss, the better the preset neural network. On this basis, the preset neural network is iteratively processed according to the obtained consistency loss. Specifically, if the obtained consistency loss is less than the preset expected value, the preset neural network is iteratively processed. By adjusting the parameters of the preset neural network after the iterative processing, the preset neural network is optimized to strengthen the consistent expression of the same defect features, make the obtained defect integrity stronger, and reduce the internal voids of the defects and edge classification errors. The parameters of the preset neural network in this embodiment include the convolution kernel used for average pooling of different scales on the target feature layer, and the number of iterations. After multiple iterations, when the obtained consistency loss reaches the preset expected value, or the number of iterations reaches a certain number, the training of the preset neural network is stopped to complete the training of the preset neural network.

[0064] Specifically, the steps after step S10 include:

[0065] Step a1, performing data enhancement processing on the defect image to obtain a transformed image, and establishing a consistent relationship between the defect image and the transformed image.

[0066] Step a2, determining whether the defect image and the transformed image meet the input requirements of a preset neural network;

[0067] Step a3: If the defect image and the transformed image do not meet the input requirements of the preset neural network, normalization and interpolation processing are performed on the defect image and the transformed image to obtain a target image.

[0068] Specifically, the steps of step S20 are as follows:

[0069] Step a4, extracting features of the target image through a preset neural network to obtain a target feature layer.

[0070] It can be seen that since the possibility of defects in products is low, the defect atlas obtained is relatively simple, and more training data is required to train the preset neural network. In this embodiment, the purpose of data enhancement on the training images is to increase the number of training images and to enhance the consistency between defects of the same type. The method of data enhancement on the training images includes rotating the defect image, adjusting brightness, transforming contrast, random cropping, mirroring and distorting the defect image to obtain different images corresponding to the same defect image (i.e., the transformed image in this embodiment). The purpose of establishing a consistent relationship between the defect image and the transformed image is to enable the preset neural network to better identify the defect image and the transformed image as different images corresponding to the same defect type to enhance the category similarity between the defect features. After obtaining the transformed image, the purpose of normalizing and interpolating the defect image and the transformed image is to make the defect image and the transformed image meet the input requirements of the preset neural network. Specifically, the method of normalization and interpolation can be to add or delete defect feature values. The input requirements of the preset neural network will be described in detail below.

[0071] Specifically, the steps after step a2 include:

[0072] Step b1: If the defect image and the transformed image meet the input requirements of the preset neural network, the defect image and the transformed image are used as the first image.

[0073] Specifically, the steps of step S20 are as follows:

[0074] Step b2: extracting features from the first image using a preset neural network to obtain a target feature layer.

[0075] It can be seen that the preset neural network in this embodiment has certain input requirements, and the role of setting the input requirements can be to normalize and unify the defect images input to the preset neural network. Specifically, when the defect images corresponding to the same defect type input to the preset neural network include defect images and their corresponding transformed images, since the transformed images may have undergone data enhancement processing such as random cropping, the sizes of the transformed images and the defect images are different. It is necessary to enlarge the transformed image to the same size as the defect image. When the image is enlarged, it is necessary to normalize and interpolate the pixel missing problem caused by the image enlargement process so that the defect image and the transformed image meet the input requirements of the preset neural network. For example, the input requirement of the preset neural network in this embodiment is that the pixel size of the input defect image is a multiple of 16. When the defect image or the transformed image does not meet this requirement, it is necessary to normalize and interpolate the defect image or the transformed image. When the image after normalization and interpolation meets this requirement, the preset neural network is used to extract features of the defect image and the transformed image to obtain the target feature layer.

[0076] Specifically, the detailed steps of step S40 include:

[0077] Step c1, obtaining a target convolution kernel used when performing average pooling of different scales on the target feature layer, and using the target convolution kernel as a parameter of the preset neural network.

[0078] Step c2, adjusting the target convolution kernel, and returning to the step of performing average pooling of different scales on the target feature layer to obtain pooling results of different scales, and obtaining the target consistency loss.

[0079] Step c3: if the target consistency loss is greater than or equal to the preset threshold, return to the step of adjusting the target convolution kernel until the target consistency loss is less than the preset threshold, thereby completing the training of the preset neural network.

[0080] It can be seen that after feature fusion of pooling results of different scales, a feature map is obtained, wherein the obtained feature map has multiple levels. The target feature map in this embodiment refers to the last layer of feature map in the above multi-level feature map. Then, according to the target feature map, the consistency loss is calculated. The obtained consistency loss represents the difference in features of different defect images corresponding to the same product defect. The parameters of the preset neural network in this embodiment include the convolution kernel used when performing average pooling of different scales on the target feature layer, that is, the target convolution kernel in this embodiment. By adjusting the target convolution kernel, the calculated consistency loss can be changed. By obtaining the change in consistency loss before and after adjusting the target convolution kernel, it is determined whether the adjustment of the target convolution kernel is accurate. It can be seen that the smaller the consistency loss obtained, the more accurate the adjustment direction of the target convolution kernel is. This cycle is iterated until the obtained consistency loss is less than the preset threshold, and the iteration is stopped to complete the training of the preset neural network.

[0081] Specifically, the steps after step S40 include:

[0082] Step d1, inputting a detection image containing a first defect into a trained preset neural network.

[0083] Step d2, obtaining a probability map of the output of the trained preset neural network.

[0084] Step d3: determining the defect type of the first defect according to the probability map.

[0085] It can be seen that when the preset neural network training is completed, the trained preset neural network can be used to perform defect detection on unknown images, wherein the unknown image may contain defects or may not contain defects. When an image that needs to be detected for defects (i.e., a detection image containing a first defect in this embodiment) is obtained, the detection image containing the first defect is input into the trained preset neural network. Unlike the training image, the detection image does not need to be pixel labeled. Instead, after inputting the trained preset neural network, the detection image is feature extracted through the preset neural network to obtain a target feature layer, and then the target feature layer is pooled to obtain a pooling result, and then Then, feature fusion is performed on the pooling result to obtain a feature map, and finally, linear operation and logistic regression operation are performed on the feature map to obtain a probability map. It can be understood that the obtained probability map represents the probability of which defect type the detection image belongs to, that is, the probability map contains defect types and the probability of the detection image belonging to each defect type, and the defect type corresponding to the highest probability value is the defect type of the first defect in this embodiment. It can be seen that if the detection image does not contain any defects, the probability of belonging to each defect type in the obtained probability map is very small. By judging whether there is a probability value greater than a preset threshold among the probability values ​​belonging to each defect type, it is determined that the detection image is an image without defects.

[0086] In this embodiment, the defect detection program based on image segmentation receives a training image containing a target defect, and performs pixel annotation on the target defect to obtain a defect image, and then performs feature extraction on the defect image through a preset neural network to obtain a target feature layer, and then performs average pooling of different scales on the target feature layer to obtain pooling results of different scales, and then performs feature fusion on the pooling results of different scales to obtain a target feature map, and calculates the consistency loss based on the target feature map, and finally completes the training of the preset neural network by iteratively processing the preset neural network and adjusting the parameters of the preset neural network after the iterative processing. Through the training of the preset neural network, richer feature representations can be obtained in the neural network training process, and the consistency expression of defects is enhanced through consistency loss, thereby improving the defect compatibility problems occurring in the detection process, increasing the integrity of defects, reducing defect classification errors, and thereby improving the stability and generalization of the neural network.

[0087] Further, refer to Figure 3 Based on the above-mentioned embodiments of the present invention, a second embodiment of the defect detection method based on image segmentation of the present invention is proposed.

[0088] This embodiment is a refinement of step S20 in the first embodiment. The difference between this embodiment and the above-mentioned embodiment of the present invention is that:

[0089] Step S21 : acquiring a target scale of the target feature layer and a scale set corresponding to the target scale, wherein the scale set includes at least two different scales.

[0090] Step S22: performing average pooling on the target feature layer according to different scales in the scale set to obtain pooling results of different scales.

[0091] It can be seen that the target scale in this embodiment refers to the scale of the target feature layer. The scales included in the scale set corresponding to the target scale may be the scale of the target feature layer (i.e., the target scale), half the scale of the target feature layer (i.e., half the target scale), and one-third the scale of the target feature layer (i.e., one-third the target scale), etc. In this embodiment, different scales correspond to different convolution kernels, and according to different scales in the scale set, these convolution kernels will be used in the process of average pooling of the target feature layer. Specifically, when the target feature layer is average pooled at one-third of the target scale, the convolution kernel corresponding to one-third of the target scale needs to be used. For example, the target feature layer is a feature matrix of 60×60, and the size of the convolution kernel used for average pooling is one-third of the target feature layer, i.e., a 20×20 grid. The result of average pooling the target feature layer using the convolution kernel is to obtain a 3×3 matrix. The matrix obtained after average pooling is the pooling result corresponding to one-third of the target scale.

[0092] Specifically, the steps of step S30 include:

[0093] Step e1, calculating the target difference between the target feature map and the target feature layer, and using the target difference as the consistency loss.

[0094] It can be seen that the target feature map is obtained by feature fusion of pooling results of different scales, and the target feature layer is the feature map of the original scale. The difference between the target feature map and the target feature layer is essentially the loss of image consistency. Therefore, by calculating the target difference between the target feature map and the target feature layer, the category loss between the defect features can be obtained, and the result is expressed as a numerical value.

[0095] In this embodiment, by performing average pooling of different scales on the target feature layer, the consistent expression of defects is enhanced, thereby improving the defect compatibility problem that occurs during the detection process, increasing the integrity of the defects, reducing defect classification errors, and further improving the stability and generalization of the neural network.

[0096] In addition, refer to Figure 4 The embodiment of the present invention further provides a defect detection device based on image segmentation, and the defect detection device based on image segmentation includes:

[0097] The pixel annotation module 10 is used to receive a training image containing a target defect, and perform pixel annotation on the target defect to obtain a defect image;

[0098] The feature extraction and pooling module 20 is used to extract features from the defect image through a preset neural network to obtain a target feature layer, and perform average pooling of different scales on the target feature layer to obtain pooling results of different scales;

[0099] A consistency loss calculation module 30 is used to perform feature fusion on the pooling results of different scales to obtain a target feature map, and calculate the consistency loss according to the target feature map;

[0100] The neural network training module 40 is used to iteratively process the preset neural network according to the consistency loss, and adjust the parameters of the preset neural network after the iterative processing to complete the training of the preset neural network.

[0101] Optionally, the defect detection device based on image segmentation includes:

[0102] A data enhancement module, used for performing data enhancement processing on the defect image to obtain a transformed image, and establishing a consistent relationship between the defect image and the transformed image;

[0103] A judgment module, used to judge whether the defect image and the transformed image meet the input requirements of a preset neural network;

[0104] A normalization and interpolation module, configured to perform normalization and interpolation processing on the defect image and the transformed image to obtain a target image if the defect image and the transformed image do not meet the input requirements of the preset neural network;

[0105] The consistency loss calculation module 30 includes:

[0106] The first feature extraction unit is used to extract features from the target image through a preset neural network to obtain a target feature layer.

[0107] Optionally, the defect detection device based on image segmentation includes:

[0108] A first image determination module, configured to use the defect image and the transformed image as a first image if the defect image and the transformed image meet the input requirements of the preset neural network;

[0109] The consistency loss calculation module 30 includes:

[0110] The second feature extraction unit is used to extract features from the first image through a preset neural network to obtain a target feature layer.

[0111] Optionally, the feature extraction and pooling module 20 includes:

[0112] A target scale acquisition unit, configured to acquire a target scale of the target feature layer and a scale set corresponding to the target scale, wherein the scale set includes at least two different scales;

[0113] The average pooling unit is used to perform average pooling on the target feature layer according to different scales in the scale set to obtain pooling results of different scales.

[0114] Optionally, the consistency loss calculation module 30 comprises:

[0115] The target difference calculation unit is used to calculate the target difference between the target feature map and the target feature layer, and use the target difference as the consistency loss.

[0116] Optionally, the neural network training module 40 includes:

[0117] A target convolution kernel acquisition unit, used to acquire a target convolution kernel used when performing average pooling of different scales on the target feature layer, and use the target convolution kernel as a parameter of the preset neural network;

[0118] An adjustment unit, used for adjusting the target convolution kernel, and returning to the step of performing average pooling of different scales on the target feature layer to obtain pooling results of different scales, and obtaining a target consistency loss;

[0119] The loop unit is used to return to the step of adjusting the target convolution kernel if the target consistency loss is greater than or equal to a preset threshold, until the target consistency loss is less than the preset threshold, thereby completing the training of the preset neural network.

[0120] Optionally, the defect detection device based on image segmentation includes:

[0121] A detection image input module, used for inputting a detection image containing a first defect into a trained preset neural network;

[0122] A probability map output module, used to obtain a probability map output by the trained preset neural network;

[0123] A defect type determination module is used to determine the defect type of the first defect according to the probability map.

[0124] In addition, an embodiment of the present invention further proposes a storage medium, on which an image segmentation-based defect detection program is stored. When the image segmentation-based defect detection program is executed by a processor, the operations in the image segmentation-based defect detection method provided in the above embodiment are implemented.

[0125] The methods executed by the above-mentioned program modules can refer to the various embodiments of the method of the present invention, and will not be described in detail here.

[0126] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity / operation / object from another entity / operation / object, and do not necessarily require or imply any such actual relationship or order between these entities / operations / objects; the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0127] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts refer to the partial description of the method embodiment. The device embodiment described above is only illustrative, and the units described as separate components may or may not be physically separated. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the present invention. Those of ordinary skill in the art can understand and implement it without paying creative work.

[0128] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0129] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, including a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the image segmentation-based defect detection method described in each embodiment of the present invention.

[0130] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A defect detection method based on image segmentation, characterized in that: The defect detection method based on image segmentation comprises the following steps: Receiving a training image containing a target defect, and annotating pixels of the target defect to obtain a defect image; By using a preset neural network, feature extraction is performed on the defect image to obtain a target feature layer, and average pooling of different scales is performed on the target feature layer to obtain pooling results of different scales; Performing feature fusion on the pooling results of different scales to obtain a target feature map, and calculating consistency loss based on the target feature map; Iteratively processing the preset neural network according to the consistency loss, and adjusting the parameters of the preset neural network after the iterative processing to complete the training of the preset neural network; Wherein, after the step of receiving a training image containing a target defect and annotating pixels of the target defect to obtain a defect image, the following steps are included: Performing data enhancement processing on the defect image to obtain a transformed image, and establishing a consistent relationship between the defect image and the transformed image; Determining whether the defect image and the transformed image meet the input requirements of a preset neural network; If the defect image and the transformed image do not meet the input requirements of the preset neural network, normalizing and interpolating the defect image and the transformed image to obtain a target image; The step of extracting features from the defect image by using a preset neural network to obtain a target feature layer comprises: The target image is subjected to feature extraction via a preset neural network to obtain a target feature layer.

2. The defect detection method based on image segmentation according to claim 1, characterized in that: After the step of determining whether the defect image and the transformed image meet the input requirements of the preset neural network, the method further comprises: If the defect image and the transformed image meet the input requirements of the preset neural network, use the defect image and the transformed image as the first image; The step of extracting features from the defect image by using a preset neural network to obtain a target feature layer comprises: The first image is subjected to feature extraction through a preset neural network to obtain a target feature layer.

3. The defect detection method based on image segmentation according to claim 1, characterized in that: The step of extracting features from the defect image by using a preset neural network to obtain a target feature layer, and performing average pooling of different scales on the target feature layer to obtain pooling results of different scales includes: Acquire a target scale of the target feature layer and a scale set corresponding to the target scale, wherein the scale set includes at least two different scales; According to different scales in the scale set, average pooling is performed on the target feature layer to obtain pooling results of different scales.

4. The defect detection method based on image segmentation according to claim 1, characterized in that: The step of performing feature fusion on the pooling results of different scales to obtain a target feature map, and calculating the consistency loss according to the target feature map includes: A target difference between the target feature map and the target feature layer is calculated, and the target difference is used as the consistency loss.

5. The defect detection method based on image segmentation according to claim 1, characterized in that: The step of iteratively processing the preset neural network according to the consistency loss and adjusting the parameters of the preset neural network after the iterative processing to complete the training of the preset neural network includes: Obtaining a target convolution kernel used when performing average pooling of different scales on the target feature layer, and using the target convolution kernel as a parameter of the preset neural network; Adjusting the target convolution kernel, and returning to the step of performing average pooling of different scales on the target feature layer to obtain pooling results of different scales, and obtaining a target consistency loss; If the target consistency loss is greater than or equal to the preset threshold, return to the step of adjusting the target convolution kernel until the target consistency loss is less than the preset threshold, thereby completing the training of the preset neural network.

6. The defect detection method based on image segmentation according to claim 1, characterized in that: After the step of iteratively processing the preset neural network according to the consistency loss and adjusting the parameters of the preset neural network after the iterative processing to complete the training of the preset neural network, the method further comprises: Inputting a detection image containing a first defect into the trained preset neural network; Obtaining a probability map of the output of the trained preset neural network; A defect type of the first defect is determined according to the probability map.

7. A defect detection device based on image segmentation, characterized in that: The defect detection device based on image segmentation comprises: A pixel annotation module is used to receive a training image containing a target defect, and perform pixel annotation on the target defect to obtain a defect image; A feature extraction and pooling module is used to extract features from the defect image through a preset neural network to obtain a target feature layer, and perform average pooling of different scales on the target feature layer to obtain pooling results of different scales; A consistency loss calculation module, used to perform feature fusion on the pooling results of different scales to obtain a target feature map, and calculate the consistency loss based on the target feature map; A neural network training module, used for iteratively processing the preset neural network according to the consistency loss, and adjusting the parameters of the preset neural network after the iterative processing to complete the training of the preset neural network; The pixel labeling module is further used to: perform data enhancement processing on the defect image to obtain a transformed image, and establish a consistent relationship between the defect image and the transformed image; determine whether the defect image and the transformed image meet the input requirements of the preset neural network; if the defect image and the transformed image do not meet the input requirements of the preset neural network, perform normalization and interpolation processing on the defect image and the transformed image to obtain a target image; The feature extraction and pooling module is also used to: extract features from the target image through a preset neural network to obtain a target feature layer.

8. A defect detection device based on image segmentation, characterized in that: The image segmentation-based defect detection device includes: a memory, a processor, and an image segmentation-based defect detection program stored in the memory and executable on the processor. When the image segmentation-based defect detection program is executed by the processor, the steps of the image segmentation-based defect detection method as described in any one of claims 1 to 6 are implemented.

9. A storage medium, characterized in that: The storage medium stores an image segmentation-based defect detection program, which, when executed by a processor, implements the steps of the image segmentation-based defect detection method according to any one of claims 1 to 6.

Images