Image defect detection method and device, computer equipment and storage medium

Abstract

The invention provides an image defect detection method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring a segmentation data set and a classification data set; training a coarse segmentation network by using the segmentation data set as a sample to obtain a trained coarse segmentation network, and processing the classification data set by using the trained coarse segmentation network to generate a corresponding pseudo segmentation label; splicing the classification data set and the pseudo segmentation labels one by one to obtain a spliced data set, inputting the spliced data set into a classification network, and training the classification network to obtain a trained classification network; and processing the segmented data set by using the trained classification network to obtain a defect positioning map. The correlation between the coarse segmentation network and the classification network is utilized, and the coarse segmentation network and the classification network are operated at the same time through bidirectional guidance. The defect detection method based on bidirectional guidance effectively improves the problems of overfitting and low precision of the segmentation network under small sample data, and can improve the performance of defect segmentation and classification.

Description

Technical Field

[0001] The present invention relates to the field of image detection technology, and in particular to an image defect detection method, device, computer equipment and storage medium. Background Art

[0002] Damage often occurs in different types of structures, such as buildings, bridges, and industrial equipment. Various types of damage are inevitable on their surfaces or important internal parts, such as wear, cracks, defects, and burns. The main goal of intelligent image damage detection is to accurately obtain damage location and characteristics to ensure the normal and safe operation of the equipment.

[0003] In the intelligent detection methods of image damage, image segmentation and classification of damage are two of the most basic and relevant tasks. Image segmentation technology can classify each pixel in the image as a certain type of damage or no damage by making pixel-level predictions on the image. Existing damage recognition networks often use U-Net networks and their variants. This network mainly adopts a codec-based structure, in which consecutive pooling or convolution steps gradually halve and restore the resolution of the feature map. Although deep learning models are popular, they are usually designed for only one task, ignoring the potential benefits of performing the two tasks jointly.

[0004] In the actual application of image defect detection technology, the training process of image segmentation network requires a large amount of image data. However, borescope acquisition and manual semantic labeling are often very time-consuming, resulting in a small number of samples in the current damage image dataset. The segmentation network is prone to overfitting during the training process. Therefore, the single use of segmentation network is not suitable for actual damage detection technology with small sample data. Summary of the invention

[0005] Based on this, it is necessary to provide an image defect detection method, device, computer equipment and storage medium to address the above technical problems.

[0006] An image defect detection method, comprising:

[0007] Acquire a segmentation data set and a classification data set, wherein the segmentation data set is an image set in which pixels of the image are annotated with preset defects, the classification data set is an image set in which types of preset defects are annotated, and the number of images in the classification data set is greater than the number of images in the segmentation data set;

[0008] Using the segmentation data set as a sample to train a coarse segmentation network to obtain a trained coarse segmentation network, and using the trained coarse segmentation network to process the classification data set to generate corresponding pseudo segmentation labels;

[0009] splicing the classification data set and the pseudo segmentation labels one by one to obtain a spliced ​​data set, inputting the spliced ​​data set into a classification network, training the classification network, and obtaining a trained classification network;

[0010] The segmented data set is processed using the trained classification network to obtain a defect location map.

[0011] In one embodiment, after the step of processing the segmented data set using the trained classification network to obtain a preset defect location map, the step further includes:

[0012] Fusing the defect localization map with image features extracted from the segmentation data set by an encoder of a strong segmentation network to obtain a fused image;

[0013] The fused image is decoded using a decoder of the strong segmentation network to obtain a defect segmentation result.

[0014] In one embodiment, the step of fusing the defect localization map with the image features extracted from the segmentation data set by the encoder of the strong segmentation network to obtain a fused image comprises:

[0015] The defect localization map is fused with the image features extracted from the segmentation data set by the encoder of the strong segmentation network through a 1x1 convolution layer, a batch normalization layer, and a ReLU activation function to obtain the fused image.

[0016] In one embodiment, the step of processing the segmented data set with the trained classification network to obtain a defect location map includes:

[0017] The segmented data set is input into the trained classification network for processing, and the defect location map is obtained through classification activation mapping.

[0018] In one embodiment, the classification data set and the pseudo segmentation labels are concatenated one by one to obtain a concatenated data set, and the concatenated data set is input into a classification network, and the classification network is trained to obtain a trained classification network:

[0019]

[0020] in, represents the classification network, F m is the output result of the classification network, I m For classification data sets,

[0021] P m is a pseudo segmentation label.

[0022] In one embodiment, the coarse segmentation network is obtained by pre-training a context encoder network on a Deepcrack dataset.

[0023] An image defect detection device, comprising:

[0024] A data set acquisition module, used to acquire a segmentation data set and a classification data set, wherein the segmentation data set is an image set with preset defects annotated on pixels of the image, the classification data set is an image set with types of preset defects annotated, and the number of images in the classification data set is greater than the number of images in the segmentation data set;

[0025] A pseudo-segmentation label generation module is used to train a coarse segmentation network using the segmentation data set as a sample to obtain a trained coarse segmentation network, and to process the classification data set using the trained coarse segmentation network to generate corresponding pseudo-segmentation labels;

[0026] A classification network training module, used for concatenating the classification data set and the pseudo segmentation labels one by one to obtain a concatenated data set, inputting the concatenated data set into the classification network, training the classification network, and obtaining a trained classification network;

[0027] The defect location map acquisition module is used to process the segmented data set using the trained classification network to obtain a defect location map.

[0028] In one embodiment, the apparatus further comprises:

[0029] A fused image acquisition module, used to fuse the defect localization map with the image features extracted from the segmentation data set by the encoder of the strong segmentation network to obtain a fused image;

[0030] The defect segmentation result obtaining module is used to decode the fused image using the decoder of the strong segmentation network to obtain the defect segmentation result.

[0031] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and wherein the processor implements the following steps when executing the computer program:

[0032] Acquire a segmentation data set and a classification data set, wherein the segmentation data set is an image set in which pixels of the image are annotated with preset defects, the classification data set is an image set in which types of preset defects are annotated, and the number of images in the classification data set is greater than the number of images in the segmentation data set;

[0033] Using the segmentation data set as a sample to train a coarse segmentation network to obtain a trained coarse segmentation network, and using the trained coarse segmentation network to process the classification data set to generate corresponding pseudo segmentation labels;

[0034] splicing the classification data set and the pseudo segmentation labels one by one to obtain a spliced ​​data set, inputting the spliced ​​data set into a classification network, training the classification network, and obtaining a trained classification network;

[0035] The segmented data set is processed using the trained classification network to obtain a defect location map.

[0036] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0037] Acquire a segmentation data set and a classification data set, wherein the segmentation data set is an image set in which pixels of the image are annotated with preset defects, the classification data set is an image set in which types of preset defects are annotated, and the number of images in the classification data set is greater than the number of images in the segmentation data set;

[0038] Using the segmentation data set as a sample to train a coarse segmentation network to obtain a trained coarse segmentation network, and using the trained coarse segmentation network to process the classification data set to generate corresponding pseudo segmentation labels;

[0039] splicing the classification data set and the pseudo segmentation labels one by one to obtain a spliced ​​data set, inputting the spliced ​​data set into a classification network, training the classification network, and obtaining a trained classification network;

[0040] The segmented data set is processed using the trained classification network to obtain a defect location map.

[0041] The above-mentioned image defect detection method, device, computer equipment and storage medium use the correlation between the coarse segmentation network and the classification network to simultaneously run the coarse segmentation network and the classification network in a bidirectional guidance manner. The coarse segmentation network provides prior guidance for the classification network to help it accurately locate and train the classification network, and then uses the trained classification network to identify the segmentation data set, thereby obtaining the location of the defects in the segmentation data set. In this way, the defect detection method based on bidirectional guidance can effectively improve the overfitting and low precision problems of the segmentation network under small sample data. The segmentation network and the classification network transfer knowledge to each other and promote each other through bidirectional guidance, which can simultaneously improve the performance of defect segmentation and classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a schematic diagram of a flow chart of an image defect detection method in one embodiment;

[0043] Figure 2 is a structural block diagram of an image defect detection device in one embodiment;

[0044] Figure 3 is an internal structure diagram of a computer device in one embodiment;

[0045] Figure 4 A schematic diagram of data flow of an image defect detection method in one embodiment;

[0046] Figure 5 The figure is a schematic diagram of the process of obtaining a damage prediction result through an image defect detection method in one embodiment. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] Embodiment 1

[0049] In this embodiment, Figure 1 As shown, a method for detecting image defects is provided, which includes:

[0050] Step 110, obtaining a segmentation data set and a classification data set, wherein the segmentation data set is an image set in which pixels of the images are annotated with preset defects, and the classification data set is an image set in which types of preset defects are annotated, and the number of images in the classification data set is greater than the number of images in the segmentation data set.

[0051] In this embodiment, a segmented data set and a classified data set are established, wherein the data volume of the classified data set is greater than the data volume of the segmented data set.

[0052] Specifically, the segmentation data set is obtained by annotating the pixels of the image one by one, and each pixel of the image is annotated with the defect type; the classification data set uses image-level annotation, and the annotation process is simpler than that of the segmentation data set. The classification data set is obtained by annotating the defects of the image. The annotation of the image in the classification data set is an overall annotation of the image, and the types of defects in the image are annotated. Since the annotation of the segmentation data set is more detailed than that of the classification data set, the number of classification data sets M is greater than the segmentation data set N, that is, M>>N. In this embodiment, the correlation between the classification network and the segmentation network is used to simultaneously improve the detection performance of the segmentation and classification networks through a two-way guidance method. In this embodiment, the defect can refer to damage or other problems in the image. The damage will be further explained as a defect below.

[0053] Specifically, the segmented dataset I n For each image X n Each pixel of is labeled, and each pixel is labeled with a type of damage or background. It is expressed as:

[0054] I n =(X n , Y n ) N

[0055] Among them, Y n is the labeled segmentation label, where y n =0 means background, y n =1,2,…,C represents different damage types, and C represents the number of damage types;

[0056] Classification Dataset I m For each image X m Image-level annotation is used. It is expressed as:

[0057] I m =(X m , Y m ) M

[0058] Y m is the classification label, where y m ∈(l1,…,l C ).

[0059] Step 120: Use the segmentation data set as a sample to train a coarse segmentation network to obtain a trained coarse segmentation network, and use the trained coarse segmentation network to process the classification data set to generate corresponding pseudo segmentation labels.

[0060] In this embodiment, a coarse segmentation network is trained on the segmentation data set, and the trained coarse segmentation network is used for the classification data set to generate corresponding pseudo segmentation labels.

[0061] In one embodiment, the coarse segmentation network is obtained by pre-training on the Deepcrack dataset using a context encoder network. In this embodiment, before using the segmentation dataset as a sample to train the coarse segmentation network, the coarse segmentation network is first obtained. In this embodiment, the coarse segmentation network is obtained by pre-training on the Deepcrack dataset using a context encoder network. After obtaining the coarse segmentation network, the segmentation dataset is used to train the coarse segmentation network.

[0062] In this embodiment, we first use the segmentation data set I with a small number of samples. nTo train a coarse segmentation network to roughly segment the damaged image, the coarse segmentation network uses a context encoder network and is pre-trained on the Deepcrack dataset. The trained coarse segmentation network is then used for a classification dataset with a large number of samples. m To generate the corresponding pseudo segmentation label P m , the obtained pseudo segmentation label P m It provides the classification network with prior information of the damage location, which can enhance the classification network's positioning and recognition capabilities. The pseudo segmentation label is expressed as:

[0063]

[0064] in, () represents the coarse segmentation network.

[0065] Step 130, concatenate the classification data set and the pseudo segmentation labels one by one to obtain a concatenated data set, input the concatenated data set into a classification network, train the classification network, and obtain a trained classification network.

[0066] In order to improve the positioning and recognition capabilities of the classification network, in this embodiment, the pseudo segmentation label P generated by the coarse segmentation network is used. m , a classification dataset I with only one image-level classification label for each image m In this embodiment, each image in the classification data set is spliced ​​with the pseudo segmentation label one by one to obtain a spliced ​​data set. m and its corresponding pseudo segmentation label P m After being concatenated, the concatenated dataset is obtained as the input of the classification network, and is input into the classification network, where the classification network adopts the popular SE-ResNeXt. It is expressed as:

[0067]

[0068] in, represents the classification network, F m Represents the output result of the classification network. The spliced ​​data obtained through the above process can effectively train the classification network and improve the recognition and positioning ability of the trained classification network for defects.

[0069] Step 140: Process the segmented data set using the trained classification network to obtain a defect location map.

[0070] In this embodiment, for segmenting data set I n Each segmented data image X n , which is fed into the classification network trained with the concatenated dataset. After being processed by the classification network, the damage localization map is obtained. It is expressed as:

[0071]

[0072] in, represents the classification activation map, Represents the refined damage localization map.

[0073] In the above embodiment, the correlation between the coarse segmentation network and the classification network is utilized, and the coarse segmentation network and the classification network are simultaneously operated in a bidirectional guidance manner. The coarse segmentation network provides a priori guidance for the classification network to help it accurately locate and train the classification network, and then the trained classification network is used to identify the segmentation data set, thereby obtaining the location of the defects in the segmentation data set. In this way, the defect detection method based on bidirectional guidance can effectively improve the overfitting and low precision problems of the segmentation network under small sample data. The segmentation network and the classification network transfer knowledge to each other and promote each other through bidirectional guidance, which can simultaneously improve the performance of defect segmentation and classification.

[0074] In one embodiment, the step of processing the segmented data set with the trained classification network to obtain a defect localization map includes: inputting the segmented data set into the trained classification network for processing, and obtaining the defect localization map through classification activation mapping.

[0075] For segmentation dataset I n Each segmented data image X n , which is fed into the classification network trained with the concatenated dataset and generates a refined damage localization map via classification activation mapping (CAM)

[0076] In one embodiment, after the step of using the trained classification network to process the segmentation data set to obtain a preset defect location map, the step also includes: fusing the defect location map with image features extracted from the segmentation data set by the encoder of the strong segmentation network to obtain a fused image; and decoding the fused image using the decoder of the strong segmentation network to obtain a defect segmentation result.

[0077] In this embodiment, the damage localization map is fused with the image features extracted from the segmentation dataset by the encoder in the strong segmentation network, and then fed to the decoder of the strong segmentation network to generate a more accurate damage segmentation result.

[0078] In one embodiment, the step of fusing the defect localization map with the image features extracted by the encoder of the strong segmentation network from the segmentation data set to obtain a fused image includes: fusing the defect localization map with the image features extracted by the encoder of the strong segmentation network from the segmentation data set through a 1x1 convolution layer, a batch normalization layer, and a ReLU activation function to obtain the fused image.

[0079] In this embodiment, in order to obtain the damage location map The refined prior information of the damage location in the segmentation process is incorporated into the segmentation process. The strong segmentation network first combines the feature map generated by the encoder with The fused feature maps are connected and passed through a 1x1 convolution layer, a batch normalization layer, and a ReLU activation function to fuse high-level semantic information and damage location information. The strong segmentation network uses a context encoder network. Finally, the fused feature map is fed to the decoder to generate a more accurate damage segmentation result.

[0080] Embodiment 2

[0081] In this embodiment, a method for detecting image defects is provided. Figure 4 ,include:

[0082] Step 1: Create a segmentation dataset and a classification dataset, where the amount of data in the classification dataset is much larger than that in the segmentation dataset.

[0083] Segmentation datasets require manual pixel-by-pixel annotation, which is often very time-consuming, while classification datasets use image-level annotation, which is simple. Therefore, the number of classification datasets M is usually much larger than the number of segmentation datasets N, that is, M>>N. We use the correlation between the classification network and the segmentation network to simultaneously improve the detection performance of the segmentation and classification networks through bidirectional guidance.

[0084] Specifically, the segmented dataset I n For each image X n It is labeled pixel by pixel, and each pixel belongs to a certain type of damage or background. It is represented as:

[0085] I n =(X n , Y n ) N

[0086] Y n is the segmentation label, where y n =0 means background, y n =1,2,…,C represents different damage types, and C represents the number of damage types;

[0087] Classification Dataset I m For each image Xm Image-level annotation is used. It is expressed as:

[0088] I m =(X m , Y m ) M

[0089] Y m is the classification label, where y m ∈(l1,…,l C ).

[0090] Then, execute step 2:

[0091] A coarse segmentation network is trained on the segmentation dataset, and the trained coarse segmentation network is used on the classification dataset to generate corresponding pseudo segmentation labels.

[0092] This embodiment first uses a segmentation dataset I with a small number of samples n To train a coarse segmentation network to roughly segment the damaged image, the coarse segmentation network uses an advanced context encoder network and is pre-trained on the Deepcrack dataset. The trained coarse segmentation network is then used for a classification dataset with a large number of samples. m To generate the corresponding pseudo segmentation label P m , the obtained pseudo segmentation label P m It provides the classification network with prior information of the damage location, which can enhance the classification network's positioning and recognition capabilities. The pseudo segmentation label is expressed as:

[0093]

[0094] in, () represents a coarse segmentation network. Figure 5 As shown, pseudo segmentation labels can be obtained.

[0095] Then, step three is performed: the images in the classification dataset are concatenated with the pseudo segmentation labels obtained in step two, and used as inputs of the classification network to train the classification network.

[0096] In order to improve the positioning and recognition capabilities of the classification network, in this example we use the pseudo segmentation label P generated by the coarse segmentation network. m , a dataset I with only one image-level classification label for each image m For training. Each classification data image X m and its corresponding pseudo segmentation label P m After being concatenated, they are used as input to the classification network, where the classification network uses the popular SE-ResNeXt. It can be expressed as:

[0097]

[0098] in represents the classification network, F m Represents the output of the classification network.

[0099] Subsequently, step four is performed: the trained classification network is used to segment the dataset to generate a refined damage localization map.

[0100] In this embodiment, for segmenting data set I n Each segmented data image X n , we feed it into the classification network and generate a refined damage localization map via classification activation mapping (CAM) It is expressed as:

[0101]

[0102] in represents the classification activation map, represents the refined damage localization map. The obtained damage localization map is as follows Figure 5 shown.

[0103] Finally, execute step 5: fuse the damage localization map obtained in step 4 with the image features extracted by the encoder in the strong segmentation network, and then feed it to the decoder of the strong segmentation network to generate a more accurate damage segmentation result.

[0104] In this embodiment, in order to obtain the damage location map in step 4 The refined prior information of the damage location in the segmentation process is incorporated into the segmentation process. The strong segmentation network first combines the feature map generated by the encoder with The fused feature maps are connected and passed through a 1x1 convolution layer, a batch normalization layer, and a ReLU activation function to fuse high-level semantic information and damage location information, where the strong segmentation network still uses the context encoder network. Finally, the fused feature map is fed to the decoder to generate a more accurate damage segmentation result.

[0105] In this embodiment, a method for intelligent image damage detection based on bidirectional guidance is proposed. This method takes into account the correlation between the segmentation network and the classification network, and uses bidirectional guidance to simultaneously run the segmentation and classification networks. The coarse segmentation network provides prior guidance for the classification network to help it accurately locate and classify damage, and then the refined positioning capability is transferred to the strong segmentation network to obtain more accurate damage segmentation. The damage detection method based on bidirectional guidance proposed in the present invention can effectively improve the problems of overfitting and low precision of the segmentation network under small sample data. The segmentation network and the classification network transfer knowledge and promote each other through bidirectional guidance, which can simultaneously improve the performance of damage segmentation and classification.

[0106] It should be understood that although Figure 1The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0107] Embodiment 3

[0108] In this embodiment, Figure 2 As shown, an image defect detection device is provided, comprising:

[0109] The data set acquisition module 210 is used to acquire a segmentation data set and a classification data set, wherein the segmentation data set is an image set in which pixels of the image are annotated with preset defects, and the classification data set is an image set in which types of preset defects are annotated, and the number of images in the classification data set is greater than the number of images in the segmentation data set;

[0110] A pseudo segmentation label generation module 220 is used to train a coarse segmentation network using the segmentation data set as a sample to obtain a trained coarse segmentation network, and to process the classification data set using the trained coarse segmentation network to generate corresponding pseudo segmentation labels;

[0111] A classification network training module 230 is used to concatenate the classification data set and the pseudo segmentation labels one by one to obtain a concatenated data set, input the concatenated data set into a classification network, train the classification network, and obtain a trained classification network;

[0112] The defect localization map acquisition module 240 is used to process the segmented data set using the trained classification network to obtain a defect localization map.

[0113] In one embodiment, the image defect detection device further includes:

[0114] A fused image acquisition module, used to fuse the defect localization map with the image features extracted from the segmentation data set by the encoder of the strong segmentation network to obtain a fused image;

[0115] The defect segmentation result acquisition module is used to decode the fused image using the decoder of the strong segmentation network to obtain the defect segmentation result.

[0116] In one embodiment, the fused image acquisition module is also used to fuse the defect localization map with the image features extracted from the segmentation data set by the encoder of the strong segmentation network through a 1x1 convolution layer, a batch normalization layer and a ReLU activation function to obtain the fused image.

[0117] In one embodiment, the defect localization map acquisition module is further used to input the segmented data set into the trained classification network for processing, and obtain the defect localization map through classification activation mapping.

[0118] In one embodiment, the classification network training module trains the classification network to obtain a trained classification network:

[0119]

[0120] in, represents the classification network, F m is the output result of the classification network, I m For classification data sets,

[0121] P m is a pseudo segmentation label.

[0122] In one embodiment, the coarse segmentation network is obtained by pre-training a context encoder network on a Deepcrack dataset.

[0123] For the specific definition of the image defect detection device, please refer to the definition of the image defect detection method above, which will not be repeated here. Each unit in the above-mentioned image defect detection device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned units can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned units.

[0124] Embodiment 4

[0125] In this embodiment, a computer device is provided. Its internal structure diagram can be shown as follows: Figure 3As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program, and the non-volatile storage medium is deployed with a database, which is used to segment data sets and classify data sets. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with other computer devices deployed with application software. When the computer program is executed by the processor, an image defect detection method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse, etc.

[0126] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0127] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0128] Step 110, obtaining a segmentation data set and a classification data set, wherein the segmentation data set is an image set in which pixels of the images are annotated with preset defects, and the classification data set is an image set in which types of preset defects are annotated, and the number of images in the classification data set is greater than the number of images in the segmentation data set.

[0129] In this embodiment, a segmented data set and a classified data set are established, wherein the data volume of the classified data set is greater than the data volume of the segmented data set.

[0130] Specifically, the segmentation data set is obtained by annotating the pixels of the image one by one, and each pixel of the image is annotated with the defect type; the classification data set uses image-level annotation, and the annotation process is simpler than that of the segmentation data set. The classification data set is obtained by annotating the defects of the image. The annotation of the image in the classification data set is an overall annotation of the image, and the types of defects in the image are annotated. Since the annotation of the segmentation data set is more detailed than that of the classification data set, the number of classification data sets M is greater than the segmentation data set N, that is, M>>N. In this embodiment, the correlation between the classification network and the segmentation network is used to simultaneously improve the detection performance of the segmentation and classification networks through a two-way guidance method. In this embodiment, the defect can refer to damage or other problems in the image. The damage will be further explained as a defect below.

[0131] Specifically, the segmented dataset I n For each image X n Each pixel of is labeled, and each pixel is labeled with a type of damage or background. It is expressed as:

[0132] I n =(X n , Y n ) N

[0133] Among them, Y n is the labeled segmentation label, where y n =0 means background, y n =1,2,…,C represents different damage types, and C represents the number of damage types;

[0134] Classification Dataset I m For each image X m Image-level annotation is used. It is expressed as:

[0135] I m =(X m , Y m ) M

[0136] Y m is the classification label, where y m ∈(l1,…,l C ).

[0137] Step 120: Use the segmentation data set as a sample to train a coarse segmentation network to obtain a trained coarse segmentation network, and use the trained coarse segmentation network to process the classification data set to generate corresponding pseudo segmentation labels.

[0138] In this embodiment, a coarse segmentation network is trained on the segmentation data set, and the trained coarse segmentation network is used for the classification data set to generate corresponding pseudo segmentation labels.

[0139] In one embodiment, the coarse segmentation network is obtained by pre-training on the Deepcrack dataset using a context encoder network. In this embodiment, before using the segmentation dataset as a sample to train the coarse segmentation network, the coarse segmentation network is first obtained. In this embodiment, the coarse segmentation network is obtained by pre-training on the Deepcrack dataset using a context encoder network. After obtaining the coarse segmentation network, the segmentation dataset is used to train the coarse segmentation network.

[0140] In this embodiment, we first use the segmentation data set I with a small number of samples. n To train a coarse segmentation network to roughly segment the damaged image, the coarse segmentation network uses a context encoder network and is pre-trained on the Deepcrack dataset. The trained coarse segmentation network is then used for a classification dataset with a large number of samples. m To generate the corresponding pseudo segmentation label P m , the obtained pseudo segmentation label P m It provides the classification network with prior information of the damage location, which can enhance the classification network's positioning and recognition capabilities. The pseudo segmentation label is expressed as:

[0141]

[0142] in, () represents the coarse segmentation network.

[0143] Step 130, concatenate the classification data set and the pseudo segmentation labels one by one to obtain a concatenated data set, input the concatenated data set into a classification network, train the classification network, and obtain a trained classification network.

[0144] In order to improve the positioning and recognition capabilities of the classification network, in this embodiment, the pseudo segmentation label P generated by the coarse segmentation network is used. m , a classification dataset I with only one image-level classification label for each image m In this embodiment, each image in the classification data set is spliced ​​with the pseudo segmentation label one by one to obtain a spliced ​​data set. m and its corresponding pseudo segmentation label P m After being concatenated, the concatenated dataset is obtained as the input of the classification network, and is input into the classification network, where the classification network adopts the popular SE-ResNeXt. It is expressed as:

[0145]

[0146] in, represents the classification network, F m Represents the output result of the classification network. The spliced ​​data obtained through the above process can effectively train the classification network and improve the recognition and positioning ability of the trained classification network for defects.

[0147] Step 140: Process the segmented data set using the trained classification network to obtain a defect location map.

[0148] In this embodiment, for segmenting data set I n Each segmented data image X n , which is fed into the classification network trained with the concatenated dataset. After being processed by the classification network, the damage localization map is obtained. It is expressed as:

[0149]

[0150] in, represents the classification activation map, Represents the refined damage localization map.

[0151] In the above embodiment, the correlation between the coarse segmentation network and the classification network is utilized, and the coarse segmentation network and the classification network are simultaneously operated in a bidirectional guidance manner. The coarse segmentation network provides a priori guidance for the classification network to help it accurately locate and train the classification network, and then the trained classification network is used to identify the segmentation data set, thereby obtaining the location of the defects in the segmentation data set. In this way, the defect detection method based on bidirectional guidance can effectively improve the overfitting and low precision problems of the segmentation network under small sample data. The segmentation network and the classification network transfer knowledge to each other and promote each other through bidirectional guidance, which can simultaneously improve the performance of defect segmentation and classification.

[0152] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0153] Fusing the defect localization map with image features extracted from the segmentation data set by an encoder of a strong segmentation network to obtain a fused image;

[0154] The fused image is decoded using a decoder of the strong segmentation network to obtain a defect segmentation result.

[0155] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0156] The defect localization map is fused with the image features extracted from the segmentation data set by the encoder of the strong segmentation network through a 1x1 convolution layer, a batch normalization layer, and a ReLU activation function to obtain the fused image.

[0157] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0158] The segmented data set is input into the trained classification network for processing, and the defect location map is obtained through classification activation mapping.

[0159] In one embodiment, the classification network is trained to obtain a trained classification network:

[0160]

[0161] in, represents the classification network, F m is the output result of the classification network, I m For classification data sets,

[0162] P m is a pseudo segmentation label.

[0163] In one embodiment, the coarse segmentation network is obtained by pre-training a context encoder network on a Deepcrack dataset.

[0164] Embodiment 5

[0165] In this embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0166] Step 110, obtaining a segmentation data set and a classification data set, wherein the segmentation data set is an image set in which pixels of the images are annotated with preset defects, and the classification data set is an image set in which types of preset defects are annotated, and the number of images in the classification data set is greater than the number of images in the segmentation data set.

[0167] In this embodiment, a segmented data set and a classified data set are established, wherein the data volume of the classified data set is greater than the data volume of the segmented data set.

[0168] Specifically, the segmentation data set is obtained by annotating the pixels of the image one by one, and each pixel of the image is annotated with the defect type; the classification data set uses image-level annotation, and the annotation process is simpler than that of the segmentation data set. The classification data set is obtained by annotating the defects of the image. The annotation of the image in the classification data set is an overall annotation of the image, and the types of defects in the image are annotated. Since the annotation of the segmentation data set is more detailed than that of the classification data set, the number of classification data sets M is greater than the segmentation data set N, that is, M>>N. In this embodiment, the correlation between the classification network and the segmentation network is used to simultaneously improve the detection performance of the segmentation and classification networks through a two-way guidance method. In this embodiment, the defect can refer to damage or other problems in the image. The damage will be further explained as a defect below.

[0169] Specifically, the segmented dataset I n For each image X n Each pixel of is labeled, and each pixel is labeled with a type of damage or background. It is expressed as:

[0170] I n =(X n , Y n ) N

[0171] Among them, Y n is the labeled segmentation label, where y n =0 means background, y n =1,2,…,C represents different damage types, and C represents the number of damage types;

[0172] Classification Dataset I m For each image X m Image-level annotation is used. It is expressed as:

[0173] I m =(X m , Y m ) M

[0174] Y m is the classification label, where y m ∈(l1,…,l C ).

[0175] Step 120: Use the segmentation data set as a sample to train a coarse segmentation network to obtain a trained coarse segmentation network, and use the trained coarse segmentation network to process the classification data set to generate corresponding pseudo segmentation labels.

[0176] In this embodiment, a coarse segmentation network is trained on the segmentation data set, and the trained coarse segmentation network is used for the classification data set to generate corresponding pseudo segmentation labels.

[0177] In one embodiment, the coarse segmentation network is obtained by pre-training on the Deepcrack dataset using a context encoder network. In this embodiment, before using the segmentation dataset as a sample to train the coarse segmentation network, the coarse segmentation network is first obtained. In this embodiment, the coarse segmentation network is obtained by pre-training on the Deepcrack dataset using a context encoder network. After obtaining the coarse segmentation network, the segmentation dataset is used to train the coarse segmentation network.

[0178] In this embodiment, we first use the segmentation data set I with a small number of samples. n To train a coarse segmentation network to roughly segment the damaged image, the coarse segmentation network uses a context encoder network and is pre-trained on the Deepcrack dataset. The trained coarse segmentation network is then used for a classification dataset with a large number of samples. m To generate the corresponding pseudo segmentation label P m , the obtained pseudo segmentation label P m It provides the classification network with prior information of the damage location, which can enhance the classification network's positioning and recognition capabilities. The pseudo segmentation label is expressed as:

[0179]

[0180] in, () represents the coarse segmentation network.

[0181] Step 130, concatenate the classification data set and the pseudo segmentation labels one by one to obtain a concatenated data set, input the concatenated data set into a classification network, train the classification network, and obtain a trained classification network.

[0182] In order to improve the positioning and recognition capabilities of the classification network, in this embodiment, the pseudo segmentation label P generated by the coarse segmentation network is used. m , a classification dataset I with only one image-level classification label for each image m In this embodiment, each image in the classification data set is spliced ​​with the pseudo segmentation label one by one to obtain a spliced ​​data set. m and its corresponding pseudo segmentation label P m After being concatenated, the concatenated dataset is obtained as the input of the classification network, and is input into the classification network, where the classification network adopts the popular SE-ResNeXt. It is expressed as:

[0183]

[0184] in, represents the classification network, F m Represents the output result of the classification network. The spliced ​​data obtained through the above process can effectively train the classification network and improve the recognition and positioning ability of the trained classification network for defects.

[0185] Step 140: Process the segmented data set using the trained classification network to obtain a defect location map.

[0186] In this embodiment, for segmenting data set I n Each segmented data image X n , which is fed into the classification network trained with the concatenated dataset. After being processed by the classification network, the damage localization map is obtained. It is expressed as:

[0187]

[0188] in, represents the classification activation map, Represents the refined damage localization map.

[0189] In the above embodiment, the correlation between the coarse segmentation network and the classification network is utilized, and the coarse segmentation network and the classification network are simultaneously operated in a bidirectional guidance manner. The coarse segmentation network provides a priori guidance for the classification network to help it accurately locate and train the classification network, and then the trained classification network is used to identify the segmentation data set, thereby obtaining the location of the defects in the segmentation data set. In this way, the defect detection method based on bidirectional guidance can effectively improve the overfitting and low precision problems of the segmentation network under small sample data. The segmentation network and the classification network transfer knowledge to each other and promote each other through bidirectional guidance, which can simultaneously improve the performance of defect segmentation and classification.

[0190] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0191] Fusing the defect localization map with image features extracted from the segmentation data set by an encoder of a strong segmentation network to obtain a fused image;

[0192] The fused image is decoded using a decoder of the strong segmentation network to obtain a defect segmentation result.

[0193] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0194] The defect localization map is fused with the image features extracted from the segmentation data set by the encoder of the strong segmentation network through a 1x1 convolution layer, a batch normalization layer, and a ReLU activation function to obtain the fused image.

[0195] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0196] The segmented data set is input into the trained classification network for processing, and the defect location map is obtained through classification activation mapping.

[0197] In one embodiment, the classification network is trained to obtain a trained classification network:

[0198]

[0199] in, represents the classification network, F m is the output result of the classification network, I m For classification data sets,

[0200] P m is a pseudo segmentation label.

[0201] In one embodiment, the coarse segmentation network is obtained by pre-training a context encoder network on a Deepcrack dataset.

[0202] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0203] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0204] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. An image defect detection method, characterized in that: include: Acquire a segmentation data set and a classification data set, wherein the segmentation data set is an image set in which pixels of the image are annotated with preset defects, the classification data set is an image set in which types of preset defects are annotated, and the number of images in the classification data set is greater than the number of images in the segmentation data set; Using the segmentation data set as a sample to train a coarse segmentation network to obtain a trained coarse segmentation network, and using the trained coarse segmentation network to process the classification data set to generate corresponding pseudo segmentation labels; splicing the classification data set and the pseudo segmentation labels one by one to obtain a spliced ​​data set, inputting the spliced ​​data set into a classification network, training the classification network, and obtaining a trained classification network; The segmented data set is processed using the trained classification network to obtain a defect location map.

2. The method according to claim 1, characterized in that After the step of using the trained classification network to process the segmented data set to obtain a preset defect location map, the following step further includes: Fusing the defect localization map with image features extracted from the segmentation data set by an encoder of a strong segmentation network to obtain a fused image; The fused image is decoded using a decoder of the strong segmentation network to obtain a defect segmentation result.

3. The method according to claim 2, characterized in that The step of fusing the defect localization map with the image features extracted from the segmentation data set by the encoder of the strong segmentation network to obtain a fused image comprises: The defect localization map is fused with the image features extracted from the segmentation data set by the encoder of the strong segmentation network through a 1x1 convolution layer, a batch normalization layer, and a ReLU activation function to obtain the fused image.

4. The method according to claim 1, characterized in that: The step of processing the segmented data set with the trained classification network to obtain a defect location map comprises: The segmented data set is input into the trained classification network for processing, and the defect location map is obtained through classification activation mapping.

5. The method according to claim 1, characterized in that The classification data set and the pseudo segmentation labels are concatenated one by one to obtain a concatenated data set, and the concatenated data set is input into the classification network, and the classification network is trained to obtain the trained classification network: in, represents the classification network, F s is the output result of the classification network, I m For classification data set, P m is a pseudo segmentation label.

6. The method according to any one of claims 1 to 5, characterized in that: The coarse segmentation network is obtained by pre-training on the Deepcrack dataset using the following encoder network.

7. An image defect detection device, characterized in that: include: A data set acquisition module, used to acquire a segmentation data set and a classification data set, wherein the segmentation data set is an image set with preset defects annotated on pixels of the image, the classification data set is an image set with types of preset defects annotated, and the number of images in the classification data set is greater than the number of images in the segmentation data set; A pseudo-segmentation label generation module is used to train a coarse segmentation network using the segmentation data set as a sample to obtain a trained coarse segmentation network, and to process the classification data set using the trained coarse segmentation network to generate corresponding pseudo-segmentation labels; A classification network training module, used for concatenating the classification data set and the pseudo segmentation labels one by one to obtain a concatenated data set, inputting the concatenated data set into the classification network, training the classification network, and obtaining a trained classification network; The defect location map acquisition module is used to process the segmented data set using the trained classification network to obtain a defect location map.

8. The device according to claim 7, characterized in that The device also includes: A fused image acquisition module, used to fuse the defect localization map with the image features extracted from the segmentation data set by the encoder of the strong segmentation network to obtain a fused image; The defect segmentation result obtaining module is used to decode the fused image using the decoder of the strong segmentation network to obtain the defect segmentation result.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images