Training method and device of defect detection model based on double-branch image reconstruction

Abstract

The application provides a training method and device for a defect detection model based on double-branch image reconstruction, wherein the method comprises the following steps: performing feature extraction on a sample abnormal image based on an initial detection model, performing image reconstruction based on the image features, performing defect detection based on the reconstructed normal image and the reconstructed abnormal image, and obtaining a defect position prediction map; performing parameter iteration on the initial detection model based on the consistency of the defect position prediction map and the sample binary image, the consistency of the reconstructed normal image and the reconstructed abnormal image on the normal area, and the consistency of the reconstructed abnormal image and the sample abnormal image on the abnormal area, and obtaining a defect detection model, which overcomes the problem that the reconstructed image and the input image are obviously different in the background area in the traditional scheme, leading to over-detection of the normal image, and the use of consistency for training can make the reconstructed images output by the model obviously different, so that the normal area is not determined as abnormal, the over-detection problem of the normal image is alleviated, and the detection accuracy is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a training method and device of a defect detection model based on double-branch image reconstruction. BACKGROUND

[0002] Industrial appearance quality inspection is an indispensable part of the industrial production process. Since manual detection is inefficient and costly, automated detection methods have emerged. Currently, deep learning-based detection methods are used to detect surface defects. However, such methods require a large number of labeled defect samples, which are difficult to collect in actual industrial settings and require a large amount of labeling cost.

[0003] Under such circumstances, unsupervised methods have been proposed. Unlike traditional supervised learning methods, unsupervised methods only require normal samples for model training, reducing the cost of sample collection and enabling detection of new defects. Currently, unsupervised defect detection methods mainly include image reconstruction-based methods. Although this method can learn the normal sample feature expression method, it still has a serious problem in normal image detection, i.e., the model cannot adapt to changes in image background information, such as differences in light angle, brightness, and clarity. These differences are removed by the model during reconstruction, resulting in significant differences between the reconstructed image and the input image in the background area, which leads to over-detection of normal images. SUMMARY

[0004] The present application provides a training method and device of a defect detection model based on double-branch image reconstruction to solve the problem of over-detection of normal images caused by the inability of the model to adapt to changes in background information, resulting in significant differences between the reconstructed image and the input image in the background area. By introducing an additional reconstruction branch to adapt to changes in the background area, the over-detection problem of normal images is alleviated by reducing the reconstruction difference in the background area. At the same time, the model training efficiency is improved, and the accuracy of defect detection by the model is optimized.

[0005] The present application provides a training method of a defect detection model based on double-branch image reconstruction, comprising:

[0006] determining a sample normal image, a sample abnormal image, and a sample binary image, the sample abnormal image being obtained by abnormal processing based on the sample normal image, and the sample binary image being used to reflect normal and abnormal areas in the sample abnormal image;

[0007] Based on the initial detection model, feature extraction is performed on the sample abnormal image, and image reconstruction is performed based on the image features obtained by feature extraction to obtain a reconstructed normal image and a reconstructed abnormal image, defect detection is performed based on the reconstructed normal image and the reconstructed abnormal image, and a defect position prediction map is obtained;

[0008] Based on the consistency of the defect position prediction map and the sample binary image, the reconstructed normal image and the reconstructed abnormal image on the normal region, and the consistency of the reconstructed abnormal image and the sample abnormal image on the abnormal region, parameter iteration is performed on the initial detection model to obtain a defect detection model.

[0009] According to the training method of the defect detection model based on the double-branch image reconstruction provided by the application, the initial detection model includes a reconstruction network and a segmentation network, and the reconstruction network includes an encoder, a first decoder and a second decoder.

[0010] The encoder is used for feature extraction on the sample abnormal image to obtain image features, the first decoder is used for image reconstruction based on the image features to obtain a reconstructed normal image, and the second decoder is used for image reconstruction based on the image features to obtain a reconstructed abnormal image.

[0011] The segmentation network is used for defect detection based on the reconstructed normal image and the reconstructed abnormal image to obtain a defect position prediction map.

[0012] According to the training method of the defect detection model based on the double-branch image reconstruction provided by the application, the training method of the defect detection model based on the double-branch image reconstruction includes:

[0013] Based on the defect position prediction map and the sample binary image, a segmentation loss is determined;

[0014] The consistency between the sample normal image and the reconstructed normal image is determined;

[0015] Based on the consistency between the sample normal image and the reconstructed normal image, the consistency of the reconstructed normal image and the reconstructed abnormal image on the normal region, and the consistency of the reconstructed abnormal image and the sample abnormal image on the abnormal region, a consistency loss is determined;

[0016] Based on the segmentation loss and the consistency loss, parameter iteration is performed on the initial detection model to obtain a defect detection model.

[0017] According to the training method of the defect detection model based on double-branch image reconstruction provided by the application, the first decoder is used for image reconstruction based on the sample mean map and the image feature to obtain a reconstructed normal image.

[0018] The sample mean map is obtained by pixel point calculation based on each sample normal image.

[0019] According to the training method of the defect detection model based on double-branch image reconstruction provided by the application, the first decoder contains multiple decoding layers, and each decoding layer is connected with a limited attention layer.

[0020] The limited attention layer is used for information screening based on the sample mean map and the decoding feature map output by the upper decoding layer to obtain a target mean map.

[0021] The decoding layer is used for decoding based on the image feature and image reconstruction based on the decoding feature map obtained by decoding and the target mean map to obtain a reconstructed normal image.

[0022] According to the training method of the defect detection model based on double-branch image reconstruction provided by the application, the limited attention layer is used for linear transformation of the sample mean map and the decoding feature map output by the upper decoding layer respectively to obtain a transformed mean map corresponding to the sample mean map and a transformed feature map corresponding to the decoding feature map output by the upper decoding layer, and information screening of the transformed mean map based on the difference degree between the transformed mean map and the transformed feature map to obtain a target mean map.

[0023] According to the training method of the defect detection model based on double-branch image reconstruction provided by the application, the transformed mean map includes a first transformed mean map and a second transformed mean map, and the first transformed mean map and the second transformed mean map are obtained by different linear transformations of the sample mean map.

[0024] The limited attention layer is used for generating an information threshold map based on the difference degree between the second transformed mean map and the transformed feature map, and information screening of the first transformed mean map based on the information threshold map to obtain a target mean map.

[0025] The application also provides a training device of a defect detection model based on double-branch image reconstruction, comprising:

[0026] A determination unit is configured to determine a sample normal image, a sample abnormal image and a sample binary image, the sample abnormal image is obtained by abnormal processing based on the sample normal image, and the sample binary image is used to reflect a normal area and an abnormal area in the sample abnormal image.

[0027] The detection unit is configured to perform feature extraction on the sample abnormal image based on an initial detection model, perform image reconstruction based on image features obtained through the feature extraction, obtain a reconstructed normal image and a reconstructed abnormal image, perform defect detection based on the reconstructed normal image and the reconstructed abnormal image, and obtain a defect position prediction map.

[0028] The training unit is configured to perform parameter iteration on the initial detection model based on consistency of the defect position prediction map, the sample binary image, the reconstructed normal image, and the reconstructed abnormal image on a normal region, and consistency of the reconstructed abnormal image and the sample abnormal image on an abnormal region, to obtain a defect detection model.

[0029] The present application also provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the training method of the defect detection model based on double-branch image reconstruction according to any one of the above when executing the program.

[0030] The present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and the computer program is executable on a processor to implement the training method of the defect detection model based on double-branch image reconstruction according to any one of the above.

[0031] The training method and device of the defect detection model based on double-branch image reconstruction provided by the present application perform feature extraction on a sample abnormal image through an initial detection model, perform image reconstruction based on image features, perform defect detection based on a reconstructed normal image and a reconstructed abnormal image, and obtain a defect position prediction map. The initial detection model is iterated based on consistency of the defect position prediction map, a sample binary image, the reconstructed normal image, and the reconstructed abnormal image on a normal region, and consistency of the reconstructed abnormal image and the sample abnormal image on an abnormal region, to obtain a defect detection model. The problem of over-detection of normal images caused by obvious differences between reconstructed images and input images on a background region in the traditional scheme is overcome. The model is trained based on consistency on the normal region and consistency on the abnormal region, which can make the differences between reconstructed images output by the model obvious, so that the normal region will not be determined as abnormal, effectively alleviating the over-detection problem of normal images, and improving the model training efficiency and detection accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0033] Figure 1 is a flowchart of a training method of a defect detection model based on double-branch image reconstruction provided by the present application;

[0034] Figure 2 is a structural schematic diagram of a reconstruction network provided by the present application;

[0035] Figure 3 is a structural schematic diagram of a limited attention layer provided by the present application;

[0036] Figure 4 is a general framework diagram of a training method of a defect detection model based on double-branch image reconstruction provided by the present application;

[0037] Figure 5 is a structural schematic diagram of a training device of a defect detection model based on double-branch image reconstruction provided by the present application;

[0038] Figure 6 is a structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0039] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without any creative work fall within the scope of protection of the present application.

[0040] Industrial appearance quality inspection occupies a pivotal position in industrial production process. Based on the feasibility, implementability, and cost of the current manual detection and supervised learning-based detection methods, unsupervised learning method becomes the mainstream. Unlike the supervised learning method, the unsupervised method only needs to collect normal samples for model training, which not only reduces the cost of sample collection, but also has the ability to detect new defects.

[0041] At present, the basic idea of the unsupervised defect detection method is as follows: first, a part of normal samples are collected, and a model is constructed to learn the normal sample data distribution. In the actual test stage, the difference between the to-be-tested sample and the normal sample data distribution learned by the model is analyzed to locate the defect area.

[0042] In the unsupervised defect detection method, the method based on image reconstruction occupies a dominant position. In the existing image reconstruction-based method, the feature expression method of the normal sample is learned through image encoding and decoding. The encoding refers to compressing the image at the original resolution into a feature map with lower resolution, and the decoding refers to decoding the extracted feature map to the original resolution and keeping it consistent with the input sample as much as possible. Since the resolution of the feature map is generally much smaller than that of the image, effective feature information can be extracted by means of this information compression and analysis. And under the ideal condition of model training, in the test stage, the model can better restore the input sample after encoding and decoding. For the abnormal sample with a defect area, the model can eliminate the influence of the defect feature in the encoding and decoding process and reconstruct a normal image. Thus, the difference between the reconstructed images before and after the reconstruction can be compared to realize the positioning of the defect area.

[0043] However, the current reconstruction type detection method still has a serious problem, that is, the reconstruction difference in the normal area of the input normal image is often significant. The reason is that the model cannot adapt to the difference in background information of the input image, such as the difference in illumination angle, brightness, definition, etc. This difference will be eliminated by the model in the reconstruction process, resulting in a significant difference between the output reconstructed image and the input image in the background area, and thus leading to over-detection of the normal image.

[0044] To this end, the present application provides a training method of a defect detection model based on double-branch image reconstruction, which aims to introduce an additional reconstruction branch in the reconstruction process, to retain the abnormal structure in the input image while reconstructing the normal image, to adapt to the information change of the background area, and to make the reconstructed normal image and the reconstructed abnormal image consistent in the normal area, and the reconstructed abnormal image consistent with the input image in the abnormal area, so as to alleviate the over-detection problem of the normal image by reducing the reconstruction difference in the background area, and thus to improve the model training efficiency and prediction accuracy. Figure 1 The present application provides a flowchart of the training method of the defect detection model based on double-branch image reconstruction, as shown in Figure 1 The method comprises the following steps:

[0045] Step 110, determining a sample normal image, a sample abnormal image and a sample binary image, the sample abnormal image being obtained by abnormal processing based on the sample normal image, and the sample binary image being used to reflect the normal area and the abnormal area in the sample abnormal image;

[0046] Specifically, before the model training is performed, the sample data required for the model iterative training needs to be obtained first. Since the embodiment of the present application is directed to the industrial appearance quality inspection in the industrial production process, i.e., applied to the defect detection of industrial products, the sample data needs to be image data, i.e., sample images. Since the contrast needs to be formed in the defect detection process and the consistency in the normal area and the abnormal area needs to be ensured respectively, the sample images need to be sample normal images without flaws / defects.

[0047] Here, the sample normal images can be normal images in various fields and various categories in industrial production, such as wood board images in board production, cotton cloth images in cloth production, component images in electronic device production, and device appearance images in medical device production, which can be collected in actual industrial production process or industrial products by image collection equipment, or can be crawled / downloaded from a specific data set (industrial product image set, industrial product database, etc.), or can be filtered from a pre-constructed training set containing a small number of normal images, which is not limited in the embodiment of the present application.

[0048] After obtaining the sample normal images, the sample abnormal images need to be determined accordingly, i.e., the sample normal images can be abnormally processed to obtain abnormal images containing flaws / defects, i.e., sample abnormal images containing abnormal areas. Here, the sample normal images can be abnormally processed by using an image augmentation method to generate artificial images with local abnormal structures, i.e., sample abnormal images with abnormal areas. At the same time, the sample binary image corresponding to the sample abnormal image can also be obtained, which can reflect the normal area and the abnormal area in the sample abnormal image. In the image, 0 represents the normal area and 1 represents the synthesized abnormal area.

[0049] Step 120, based on the initial detection model, performing feature extraction on the sample abnormal image, and performing image reconstruction based on the image features obtained by feature extraction, to obtain a reconstructed normal image and a reconstructed abnormal image, and performing defect detection based on the reconstructed normal image and the reconstructed abnormal image to obtain a defect position prediction map;

[0050] Specifically, after obtaining the sample abnormal image through step 110, step 120 can be executed to perform defect detection on the sample abnormal image by the initial detection model to obtain the defect position in the image. This process can specifically include:

[0051] Firstly, the sample abnormal image can be subjected to feature extraction by the initial detection model, so as to obtain the image features of the sample abnormal image. Specifically, the sample abnormal image can be input into the initial detection model, and the initial image reconstruction model can extract the features of the input sample abnormal image, so as to extract the image texture, semantics, details and other information contained therein, thereby obtaining the image features. Here, the feature extraction process for the sample abnormal image is actually the encoding of the image by the encoder, so as to compress the sample abnormal image at the original resolution into a feature map with lower resolution, so as to better extract the effective feature information, thereby obtaining the image features.

[0052] Then, the initial detection model can be used to reconstruct the image according to the image features obtained by feature extraction, so as to obtain the reconstructed normal image and the reconstructed abnormal image. That is, the image can be reconstructed according to the image features of the sample abnormal image encoded by the encoder, so as to reconstruct the normal image and the abnormal image. The normal image is an image that does not contain the abnormal region in the input sample abnormal image, that is, the abnormal structure in the sample abnormal image is erased during the reconstruction of the image, thereby obtaining the reconstructed normal image. Unlike the normal image, the abnormal image contains the abnormal region in the input sample abnormal image, that is, the abnormal structure in the input sample abnormal image is retained during the reconstruction of the image, thereby obtaining the reconstructed abnormal image.

[0053] Here, the image reconstruction process according to the image features can be understood as a feature decoding process based on the decoder, that is, the extracted image features are decoded to the original resolution, so that the decoded normal image and abnormal image are as consistent as possible in the normal region, and the decoded abnormal image and the sample abnormal image are as consistent as possible in the abnormal region. In this way, when comparing the differences between the two, the model can avoid determining the normal region as abnormal, thereby effectively alleviating the over-detection problem when the model detects defects in the normal image.

[0054] Then, the reconstructed normal image and the reconstructed abnormal image can be used for defect detection to obtain the defect position in the input sample abnormal image. That is, the initial detection model can be used for defect detection based on the reconstructed normal image and the reconstructed abnormal image, so as to locate the defect position in the sample abnormal image, thereby obtaining the defect position prediction map. Here, the differences between the reconstructed normal image and the reconstructed abnormal image can be captured by the initial detection model, and the different abnormal regions in the two images can be predicted by analyzing the differences between the two images, thereby locating the defect region corresponding to the input sample abnormal image, that is, the defect position prediction map can be obtained. Here, the defect position prediction map is a pixel-level abnormal position prediction map, and therefore it can also be called a pixel-level defect position prediction map.

[0055] At step 130, based on the consistency of the reconstructed normal image and the reconstructed abnormal image on the normal region, and the consistency of the reconstructed abnormal image and the sample abnormal image on the abnormal region, the parameter iteration is performed on the initial detection model based on the defect position prediction map and the sample binary image, to obtain a defect detection model.

[0056] Specifically, after obtaining the defect position prediction map, the model training can be performed in combination with the reconstructed normal image and the reconstructed abnormal image, and the sample binary image, to obtain the defect detection model. The process can specifically include:

[0057] During the image reconstruction, when reconstructing the normal image, the target is to eliminate the abnormal structure in the input sample abnormal image, to obtain the reconstructed normal image that is completely normal and similar to the sample normal image, that is, by repairing the abnormal region in the sample abnormal image as much as possible, to obtain the reconstructed normal image that is completely normal. Correspondingly, when reconstructing the abnormal image, the target is to retain the abnormal structure in the input sample abnormal image as much as possible, to obtain the reconstructed abnormal image containing the abnormal region, that is, to obtain the reconstructed abnormal image whose abnormal region is consistent with the abnormal region in the sample abnormal image, while the consistency of the normal region in the reconstructed normal image and the reconstructed abnormal image derived from the input sample abnormal image needs to be maintained.

[0058] In view of this, in the embodiment of the present application, when measuring the loss function of the model in the overall process of defect detection of the initial detection model, the above two consistencies can be considered to measure the specific loss of the model in the image reconstruction process, that is, the loss of the initial detection model in the coding and decoding process can be judged by the consistency of the reconstructed normal image and the reconstructed abnormal image on the normal region, and the consistency of the reconstructed abnormal image and the sample abnormal image on the abnormal region.

[0059] Specifically, the pixel difference of the reconstructed normal image and the reconstructed abnormal image on the normal region, and the pixel difference of the reconstructed abnormal image and the sample abnormal image on the abnormal region can be compared, to determine the specific loss of the initial detection model on the two consistencies. Here, the difference between the images at the pixel level can be judged by the mean square error.

[0060] At the same time, considering that the model not only contains the feature extraction and image reconstruction process corresponding to the coding and decoding, but also contains the defect detection process based on the reconstructed image, therefore, the loss measurement can also be based on the predicted defect position prediction map, that is, the loss judgment can be performed in combination with the defect position prediction map and the sample binary image corresponding to the sample abnormal image, specifically, the difference between the defect position prediction map and the sample binary image is compared to obtain the loss in the defect detection.

[0061] Further, the initial detection model can be trained according to the loss of the coding process and the loss in the defect detection process, that is, the consistency of the defect position prediction map and the sample binary image, the reconstructed normal image and the reconstructed normal region of the abnormal image, and the consistency of the reconstructed abnormal image and the sample abnormal image in the abnormal region, the model loss is determined, the parameters are adjusted according to the model loss, so that the adjusted initial detection model can make the output reconstructed normal image and the reconstructed abnormal image as consistent as possible in the normal region when facing the input abnormal image, and make the output reconstructed abnormal image and the input sample abnormal image as consistent as possible in the abnormal region, in short, make the normal regions of different reconstructed images output by the model as consistent as possible, and make the abnormal regions of the output abnormal image and the input abnormal image as consistent as possible, and finally a trained defect detection model can be obtained.

[0062] In the embodiment of the application, while the normal image is reconstructed by the image features, the reconstructed abnormal image retaining the abnormal structure in the input sample abnormal image is also reconstructed, and the model is trained by the consistency of the reconstructed normal image and the reconstructed abnormal image in the normal region and the consistency of the reconstructed abnormal image and the sample abnormal image in the abnormal region, so that the model can adapt to the information change of the background region and will not determine the normal region in the image as abnormal when capturing the difference between the two for defect detection, thereby effectively alleviating the over-detection problem of the normal image, and further improving the model training efficiency and detection accuracy.

[0063] The training method of the defect detection model based on the double-branch image reconstruction provided by the application extracts features from the sample abnormal image by the initial detection model, reconstructs the image based on the image features, detects defects based on the reconstructed normal image and the reconstructed abnormal image, and obtains a defect position prediction map; the initial detection model is iterated based on the consistency of the defect position prediction map and the sample binary image, the reconstructed normal image and the reconstructed abnormal image in the normal region, and the consistency of the reconstructed abnormal image and the sample abnormal image in the abnormal region, to obtain a defect detection model, which overcomes the problem of obvious difference between the reconstructed image and the input image in the background region in the traditional scheme, leading to over-detection of the normal image, and uses the consistency in the normal region and the consistency in the abnormal region for model training, so that the difference between the reconstructed images output by the model is obvious, thereby avoiding determining the normal region as abnormal, effectively alleviating the over-detection problem of the normal image, and improving the model training efficiency and detection accuracy.

[0064] Based on the above embodiment, the initial detection model includes a reconstruction network and a segmentation network, the reconstruction network includes an encoder, a first decoder and a second decoder;

[0065] The encoder is used for feature extraction on the sample abnormal image to obtain an image feature;

[0066] The first decoder is used for image reconstruction based on the image feature to obtain a reconstructed normal image;

[0067] The second decoder is used for image reconstruction based on the image feature to obtain a reconstructed abnormal image;

[0068] The segmentation network is used for defect detection based on the reconstructed normal image and the reconstructed abnormal image to obtain a defect position prediction map.

[0069] It is considered that in the auto-encoder of the traditional reconstruction type method, with the increase of the network layers, the low-level semantic information such as texture contained on the feature map is gradually lost, that is, the high-level semantic information of the image is mainly contained on the deep feature map, which makes the reconstructed image possibly different from the input image in the texture and other details of the normal area, and when this difference can be captured by the model, the normal area will also be judged as abnormal, thereby the problem of over-detection of the normal image occurs.

[0070] Based on this, in the embodiment of the present application, when the image is reconstructed by the initial detection model, the normal image is reconstructed at the same time, and the abnormal image containing the abnormal structure in the input sample abnormal image is also reconstructed, that is, an additional reconstruction branch is introduced to reconstruct the abnormal image. In short, based on the consideration of the problems existing in the structure of the auto-encoder, a double-branch auto-encoder structure is designed in the embodiment of the present application to overcome the problem of erroneous judgment existing in the traditional scheme.

[0071] Specifically, in the embodiment of the present application, the initial detection model contains two sub-networks, that is, a reconstruction network and a segmentation network, wherein the reconstruction network is used for image reconstruction based on the input sample abnormal image, and the segmentation network is used for defect detection according to the reconstructed normal image and the reconstructed abnormal image output by the reconstruction network to obtain a defect position prediction map.

[0072] Figure 2 is a structural schematic diagram of the reconstruction network provided by the present application, as Figure 2 shown, the reconstruction network adopts a double-branch auto-encoder structure, which includes an encoder, a first decoder and a second decoder; wherein the encoder is used for feature extraction on the input sample abnormal image to obtain an image feature, that is, the encoder can compress the sample abnormal image under the original resolution into a feature map with lower resolution to better extract the effective feature information therein, thereby obtaining the image feature; and the decoder is used for image reconstruction according to the image feature to obtain a reconstructed image.

[0073] Specifically, the first decoder is configured to reconstruct a normal image according to the image features to obtain a reconstructed normal image, and the second decoder is configured to reconstruct an abnormal image according to the image features to obtain a reconstructed abnormal image. The reconstructed normal image does not contain the abnormal region in the input sample abnormal image, and the reconstructed abnormal image contains the abnormal region and has a normal region consistent with the reconstructed normal image. That is, the reconstructed normal image and the reconstructed abnormal image are consistent in the normal region and only differ in the abnormal region, so that in subsequent comparison, the segmentation network will not determine the normal region as abnormal, thereby effectively alleviating the over-detection problem.

[0074] Based on the above embodiment, step 130 comprises:

[0075] Based on the defect position prediction map and the sample binary image, a segmentation loss is determined.

[0076] The consistency between the sample normal image and the reconstructed normal image is determined.

[0077] Based on the consistency between the sample normal image and the reconstructed normal image, the consistency of the reconstructed normal image and the reconstructed abnormal image in the normal region, and the consistency of the reconstructed abnormal image and the sample abnormal image in the abnormal region, a consistency loss is determined.

[0078] Based on the segmentation loss and the consistency loss, the initial detection model is iterated in parameters to obtain a defect detection model.

[0079] Specifically, in step 130, according to the defect position prediction map and the sample binary image, the consistency of the reconstructed normal image and the reconstructed abnormal image in the normal region, and the consistency of the reconstructed abnormal image and the sample abnormal image in the abnormal region, the process of iterating the initial detection model in parameters to obtain the defect detection model can specifically comprise:

[0080] First, the loss of the model in defect detection can be measured according to the defect position prediction map obtained by the defect detection of the segmentation model and the sample binary image corresponding to the sample abnormal image, that is, by comparing the defect position prediction map and the sample binary image, the difference between the two can be determined, and the loss of the model, i.e., the segmentation loss, can be determined according to the difference;

[0081] At the same time, the loss of the model in image reconstruction can be measured according to the reconstructed normal image and the reconstructed abnormal image obtained by the image reconstruction of the reconstruction network, and the sample normal image and the sample abnormal image, that is, the loss of the model in the reconstruction process, i.e., the consistency loss, can be measured by the consistency between the sample normal image and the reconstructed normal image, the consistency of the sample abnormal image and the reconstructed abnormal image in the abnormal region, and the consistency of the reconstructed normal image and the reconstructed abnormal image in the normal region;

[0082] Here, the consistency loss of the initial detection model can be obtained by comparing the pixel difference between the normal sample image and the reconstructed normal image, the pixel difference between the reconstructed normal image and the reconstructed abnormal image on the normal region, and the pixel difference between the reconstructed abnormal image and the sample abnormal image on the abnormal region.

[0083] Then, the model training can be performed according to the segmentation loss and the consistency loss to obtain the trained defect detection model. That is, the initial detection model can be iterated based on the segmentation loss and the consistency loss to obtain the defect detection model. Here, the parameters are adjusted in combination with the two different levels of loss, so that the normal regions of different reconstructed images output by the adjusted model are as consistent as possible, the abnormal regions between the output abnormal image and the input abnormal image are as consistent as possible, and the normal image output and the sample normal image are as consistent as a whole. Finally, the trained defect detection model can be obtained.

[0084] The model training based on the consistency between the sample normal image and the reconstructed normal image can make the initial detection model (the first decoder) have the ability to reconstruct a complete normal image, so that the reconstructed normal image output by the initial detection model can be as close as possible to the sample normal image, and the reconstruction ability of the model is improved. The reconstructed normal image and the reconstructed abnormal image are consistent on the normal region, so that the initial detection model (the second decoder) can adapt to the information change of the background region during the training process, and has the ability to reconstruct the normal background. Correspondingly, the reconstructed abnormal image and the sample abnormal image are consistent on the abnormal region, so that the initial detection model (the second decoder) has the ability to retain the abnormal structure in the sample abnormal image during the training process.

[0085] In the embodiment of the present application, while reconstructing the normal image, an additional reconstruction branch is introduced to reconstruct the reconstructed abnormal image retaining the abnormal structure in the sample abnormal image, and the sample normal image and the reconstructed normal image are consistent as a whole, the reconstructed normal image and the reconstructed abnormal image are consistent on the normal region, and the reconstructed abnormal image is consistent with the input image on the abnormal region. Thus, the over-detection problem of the normal image is alleviated by reducing the reconstruction difference of the background region, and the training efficiency and prediction accuracy of the model are improved.

[0086] Based on the above embodiment, the calculation formula of the consistency loss can be represented as:

[0087] L = L1 + L2 + L3

[0088] L1 = L r (I rn ,I ori )

[0089] L2=L r ((1-M)⊙I ri ,(1-M)⊙I rn )

[0090] L3=L r (M⊙I ri ,M⊙I in )

[0091] Wherein, L represents consistency loss, L1, L2 and L3 respectively represent consistency based on sample normal image and reconstructed normal image, reconstructed normal image and reconstructed abnormal image keep consistent on normal region, and reconstructed abnormal image and sample abnormal image keep consistent on abnormal region, and the loss determined.

[0092] L r (I rn ,I ori ) represents the MSE (Mean Square Error, Mean Square Error) between the reconstructed normal image I rn and the sample normal image I ori , here the difference between the images can be measured by the mean square error at the pixel level, the smaller the MSE, the smaller the difference, the better the consistency, and vice versa. L r ((1-M)⊙I ri ,(1-M)⊙I rn ) represents the mean square error of the reconstructed abnormal image I ri and the reconstructed normal image I rn on the normal region, (1-M) represents the normal region in the two. L r (M⊙I ri ,M⊙I in ) represents the mean square error of the reconstructed abnormal image I ri and the sample abnormal image I in on the abnormal region, M represents the abnormal region. ⊙ represents the pixel-by-pixel multiplication operation.

[0093] Based on the above embodiment, the first decoder is used for image reconstruction based on the sample mean map and the image features to obtain the reconstructed normal image.

[0094] The sample mean map is calculated based on each sample normal image.

[0095] In addition to the above-mentioned problem that the model cannot adapt to the information change of the background area, resulting in a significant difference between the reconstructed image and the input image in the background area, and further causing the problem of over-detection of normal images, there is another very serious problem, i.e., the problem of missing detection of abnormal images. Specifically, after the input abnormal image is processed by the reconstruction model, the defect area is often not completely eliminated, and a part of the abnormal structure is still left in the reconstructed image. These residual abnormal structures will affect the subsequent defect detection process based on the reconstruction difference, so that the model is difficult to capture the difference, or the captured difference between the images before and after reconstruction is small, thereby causing the model to mistakenly determine the abnormal image as normal, and further causing the problem of missing detection of abnormal images.

[0096] To this end, in the embodiments of the present application, the feature information of the sample normal image is introduced in the reconstruction process as prior information to assist the model in image reconstruction, so as to ensure that there is no abnormal structure in the reconstructed normal image as much as possible, thereby reducing the risk of missing detection. That is, the feature information of the sample normal image is introduced into the first decoder of the double-branch autoencoder of the reconstruction network, so as to improve the repair ability of the first decoder for abnormal areas.

[0097] Specifically, when reconstructing the image according to the image feature to reconstruct the normal image, the first decoder can refer to the feature information of the sample normal image, reconstruct the complete normal image according to the image feature, and obtain the reconstructed normal image. Here, the feature information of the sample normal image can be understood as the mean image of the sample normal image, i.e., the sample mean image, which can be obtained by mean calculation of all sample normal images, i.e., calculating the mean of all sample normal images pixel by pixel, thereby obtaining the sample mean image.

[0098] That is, the image can be directly reconstructed according to the sample mean image and the image feature to repair the abnormal area in the sample abnormal image in the reconstruction process, so as to reconstruct the reconstructed normal image as consistent as possible with the sample normal image, avoid the residual of the abnormal structure in the input sample abnormal image, and further enable the model to effectively capture the difference between the reconstructed images in the abnormal area when performing defect detection based on the reconstructed normal image and the reconstructed abnormal image, thereby effectively avoiding the problem of missing detection caused by determining the abnormal area as normal. Through the repair of the abnormal area in the reconstruction process, the effective alleviation of the abnormal missing detection problem is realized.

[0099] Based on the above embodiments, the first decoder comprises a plurality of decoding layers, and each decoding layer is connected with a limited attention layer;

[0100] The limited attention layer is used for information screening based on the sample mean image and the decoding feature map output by the upper decoding layer to obtain a target mean image;

[0101] The decoding layer is used for decoding based on the image features, and reconstructing the image based on the decoded feature map and the target mean value map obtained by decoding to obtain the reconstructed normal image.

[0102] Specifically, in the process of reconstructing the image based on the sample mean value map of the reference sample normal image, it is considered that there may be information irrelevant to the reconstruction in the sample mean value map, or information not required by the first decoder when reconstructing the reconstructed normal image. In short, not all information in the sample mean value map is required by the first decoder. Therefore, in order to reduce the inflow of information and ensure the relevance and effectiveness of the inflow information, a gated attention unit (GA) is arranged in the embodiment of the present application. The mechanism can filter the prior information and input the information required for anomaly repair to the first decoder.

[0103] The first decoder includes a plurality of decoding layers, and a gated attention layer (gated attention mechanism) is arranged after each decoding layer. The decoding layer can decode the image features to obtain the decoded features, i.e., the decoded image features, and the gated attention layer connected after the decoding layer can filter the information based on the sample mean value map and the decoded feature map output by the upper decoding layer to obtain the information required for image reconstruction, and obtain the target mean value map. Then, the decoding layer can reconstruct based on the target mean value map and the decoded feature map output by the upper decoding layer, and finally obtain the reconstructed normal image output by the first decoder.

[0104] Based on the above embodiment, the gated attention layer is used for linearly transforming the sample mean value map and the decoded feature map output by the upper decoding layer respectively to obtain the transformed mean value map corresponding to the sample mean value map and the transformed feature map corresponding to the decoded feature map output by the upper decoding layer. The information of the transformed mean value map is filtered based on the difference between the transformed mean value map and the transformed feature map to obtain the target mean value map.

[0105] Specifically, Figure 3 is a structural schematic diagram of the gated attention layer provided by the present application, as Figure 3 shown, when the gated attention layer filters the information based on the sample mean value map and the decoded feature map output by the upper decoding layer to obtain the target mean value map, it can first linearly transform the decoded feature map x output by the upper decoding layer and the sample mean value map μ of the sample normal image respectively, so as to capture the relationship between the two in the subsequent process, thereby obtaining the transformed mean value map corresponding to the sample mean value map and the transformed feature map corresponding to the decoded feature map output by the upper decoding layer.

[0106] Then, the limited attention layer can measure the difference between the linearly transformed feature maps and perform information screening based on the difference to obtain the target mean value map, that is, the difference between the transformed mean value map and the transformed feature map can be determined, and the transformed mean value map is screened based on the difference between the two to screen the information required for image reconstruction, thereby obtaining the target mean value map.

[0107] Based on the above embodiment, the transformed mean value map includes a first transformed mean value map and a second transformed mean value map, and the first transformed mean value map and the second transformed mean value map are obtained by different linear transformations of the sample mean value map.

[0108] The limited attention layer is used to generate an information threshold map based on the difference between the second transformed mean value map and the transformed feature map, and the first transformed mean value map is screened based on the information threshold map to obtain the target mean value map.

[0109] Specifically, when the sample mean value map μ is linearly transformed to obtain a transformed mean value map, specifically, the sample mean value map μ is linearly transformed to obtain different transformed mean value maps, that is, the sample mean value map μ is respectively linearly transformed by f v and f k , to obtain a first transformed mean value map v and a second transformed mean value map k. The process of linearly transforming the decoded feature map x to obtain the transformed feature map is to linearly transform the decoded feature map x by f q , to obtain a transformed feature map q.

[0110] On this basis, when the limited attention layer screens the transformed mean value map based on the difference between the transformed mean value map and the transformed feature map to obtain the target mean value map, the difference between the second transformed mean value map k and the transformed feature map q in the transformed mean value map can be measured first to determine the information required for image reconstruction by the first decoder, and an information threshold map is generated, and then the information on the sample mean value map can be screened through the information threshold map to input the prior information required for anomaly repair into the first decoder, so that the first decoder has better ability to repair the abnormal area in the input sample abnormal image.

[0111] Here, when calculating the difference between the transformed feature map q and the second transformed mean value map k, the limited attention layer adopts a method different from the self-attention mechanism in the Transformer, because the matrix multiplication in the self-attention mechanism occupies a large amount of video memory when calculating, which requires a higher computing device, and in most cases, the computing device is often difficult to meet this computing requirement, therefore, in order to reduce the video memory occupation, the limited attention layer in the embodiment of the application adds the transformed feature map q and the second transformed mean value map k on the channel and inputs them into the convolution layer f cIn order to obtain the relationship between the two, an information threshold map thr is generated, and the value on thr is limited by the sigmoid function in the attention limiting layer to be within [0, 1].

[0112] After that, the attention limiting layer can perform information screening on the first transformed mean value map according to the information threshold map, so as to obtain the target mean value map, that is, the information threshold map thr is multiplied on the first transformed mean value map v to control the information flowing into the first decoder at each pixel point in the first mean value map, so as to obtain the screened target mean value map. Thereafter, the target mean value map can be added to the decoding feature map x output by the upper decoding layer to obtain the output feature map out and take the output feature map out as the input information of the lower decoding layer, so that the introduction of the prior information is completed.

[0113] In addition, in the embodiment of the present application, a limiting attention layer is arranged after each decoding layer in the first decoder. After the introduction of the prior information by the limiting attention layer, the sufficiency, integrity and effectiveness of the information flow can be guaranteed, which provides important and irreplaceable help for the repair of the abnormal area in the image reconstruction process.

[0114] Figure 4 The training method of the defect detection model based on the double-branch image reconstruction provided by the present application is shown in the overall framework diagram as shown in Figure 4 The method comprises the following steps.

[0115] Firstly, a sample normal image, a sample abnormal image and a sample binary image are determined, the sample abnormal image is obtained based on abnormal processing of the sample normal image, and the sample binary image is used to reflect the normal area and the abnormal area in the sample abnormal image.

[0116] Then, based on an initial detection model, image features are extracted from the sample abnormal image, and based on the image features obtained by the feature extraction, a reconstructed normal image and a reconstructed abnormal image are obtained by image reconstruction, and a defect position prediction map is obtained by defect detection based on the reconstructed normal image and the reconstructed abnormal image; wherein the initial detection model comprises a reconstruction network and a segmentation network, the reconstruction network comprises an encoder, a first decoder and a second decoder; the encoder is used to extract image features from the sample abnormal image, the first decoder is used to reconstruct the normal image based on the image features, and the second decoder is used to reconstruct the abnormal image based on the image features; the segmentation network is used to detect defects based on the reconstructed normal image and the reconstructed abnormal image to obtain the defect position prediction map.

[0117] Subsequently, based on the defect position prediction map and the sample binary map, a segmentation loss is determined; consistency between the sample normal image and the reconstructed normal image is determined; based on the consistency between the sample normal image and the reconstructed normal image, the consistency of the reconstructed normal image and the reconstructed abnormal image on the normal region, and the consistency of the reconstructed abnormal image and the sample abnormal image on the abnormal region, a consistency loss is determined; based on the segmentation loss and the consistency loss, the initial detection model is iterated in parameters to obtain the defect detection model.

[0118] Further, the first decoder comprises a plurality of decoding layers, and each decoding layer is connected with a limited attention layer; the limited attention layer is used for respectively performing linear transformation on the sample mean map and the decoding feature map output by the upper decoding layer to obtain a transformed mean map corresponding to the sample mean map and a transformed feature map corresponding to the decoding feature map output by the upper decoding layer, generating an information threshold map based on the difference degree between the second transformed mean map and the transformed feature map in the transformed mean map, and performing information screening on the first transformed mean map based on the information threshold map to obtain a target mean map. Here, the sample mean map is obtained by pixel point calculation based on each sample normal image; the transformed mean map comprises a first transformed mean map and a second transformed mean map, and the first transformed mean map and the second transformed mean map are obtained by different linear transformations on the sample mean map. The decoding layer is used for decoding based on image features and reconstructing an image based on the decoding feature map obtained by decoding and the target mean map to obtain a reconstructed normal image.

[0119] The method provided by the embodiment of the present application extracts features of the sample abnormal image through the initial detection model, reconstructs an image based on the image features, detects defects based on the reconstructed normal image and the reconstructed abnormal image to obtain a defect position prediction map; iterates the initial detection model in parameters based on the defect position prediction map and the sample binary map, the consistency of the reconstructed normal image and the reconstructed abnormal image on the normal region, and the consistency of the reconstructed abnormal image and the sample abnormal image on the abnormal region to obtain the defect detection model, which overcomes the problem that the reconstructed image and the input image have obvious differences on the background region in the traditional scheme, leading to over-detection of the normal image. The model training is performed using the consistency on the normal region and the consistency on the abnormal region, which can make the reconstructed images output by the model have obvious differences, so that the normal region will not be determined as abnormal, effectively alleviating the over-detection problem of the normal image, and improving the model training efficiency and the detection accuracy.

[0120] The training device of the defect detection model based on double-branch image reconstruction provided by the present application is described below, and the training device of the defect detection model based on double-branch image reconstruction described below can be correspondingly referred to the training method of the defect detection model based on double-branch image reconstruction described above.

[0121] Figure 5is a structural schematic diagram of a training device of a defect detection model based on double-branch image reconstruction provided by the present application, as shown in the figure, the device comprises: Figure 5

[0122] A determination unit 510 is configured to determine a sample normal image, a sample abnormal image, and a sample binary image, the sample abnormal image being obtained by performing abnormal processing on the sample normal image, and the sample binary image being used to reflect normal regions and abnormal regions in the sample abnormal image.

[0123] A detection unit 520 is configured to perform feature extraction on the sample abnormal image based on an initial detection model, perform image reconstruction based on image features obtained through feature extraction, obtain a reconstructed normal image and a reconstructed abnormal image, perform defect detection based on the reconstructed normal image and the reconstructed abnormal image, and obtain a defect position prediction map.

[0124] A training unit 530 is configured to perform parameter iteration on the initial detection model based on consistency of the defect position prediction map and the sample binary image, consistency of the reconstructed normal image and the reconstructed abnormal image on normal regions, and consistency of the reconstructed abnormal image and the sample abnormal image on abnormal regions, and obtain a defect detection model.

[0125] The training device of the defect detection model based on double-branch image reconstruction provided by the present application performs feature extraction on a sample abnormal image through an initial detection model, performs image reconstruction based on image features, performs defect detection based on a reconstructed normal image and a reconstructed abnormal image, and obtains a defect position prediction map. Parameter iteration is performed on the initial detection model based on consistency of the defect position prediction map and a sample binary image, consistency of the reconstructed normal image and the reconstructed abnormal image on normal regions, and consistency of the reconstructed abnormal image and the sample abnormal image on abnormal regions, and a defect detection model is obtained. The problem that there is a significant difference between reconstructed images and input images in the background region in the traditional scheme, leading to over-detection of normal images, is overcome. Model training is performed using consistency on normal regions and consistency on abnormal regions, which can make the difference between reconstructed images output by the model obvious, so that normal regions will not be determined as abnormal, effectively alleviating the over-detection problem of normal images, and improving the model training efficiency and detection accuracy.

[0126] Based on the above embodiment, the initial detection model comprises a reconstruction network and a segmentation network, the reconstruction network comprises an encoder, a first decoder, and a second decoder;

[0127] The encoder is configured to perform feature extraction on the sample abnormal image to obtain image features, the first decoder is configured to perform image reconstruction based on the image features to obtain a reconstructed normal image, and the second decoder is configured to perform image reconstruction based on the image features to obtain a reconstructed abnormal image.​

[0128] The segmentation network is used for defect detection based on the reconstructed normal image and the reconstructed abnormal image to obtain a defect position prediction map.

[0129] Based on the above embodiment, the training unit 530 is configured to:

[0130] Based on the defect position prediction map and the sample binary map, a segmentation loss is determined.

[0131] Consistency between the sample normal image and the reconstructed normal image is determined.

[0132] Based on the consistency between the sample normal image and the reconstructed normal image, the consistency of the reconstructed normal image and the reconstructed abnormal image on the normal region, and the consistency of the reconstructed abnormal image and the sample abnormal image on the abnormal region, a consistency loss is determined.

[0133] Based on the segmentation loss and the consistency loss, the initial detection model is iterated in parameters to obtain a defect detection model.

[0134] Based on the above embodiment, the first decoder is configured to perform image reconstruction based on a sample mean map and the image feature to obtain a reconstructed normal image.

[0135] The sample mean map is obtained by pixel point calculation based on each sample normal image.

[0136] Based on the above embodiment, the first decoder comprises a plurality of decoding layers, and each decoding layer is connected with a limited attention layer.

[0137] The limited attention layer is configured to perform information screening based on the sample mean map and a decoding feature map output by an upper decoding layer to obtain a target mean map.

[0138] The decoding layer is configured to perform decoding based on the image feature, and perform image reconstruction based on a decoded feature map obtained by decoding and the target mean map to obtain a reconstructed normal image.

[0139] Based on the above embodiment, the limited attention layer is configured to perform linear transformation on the sample mean map and the decoding feature map output by the upper decoding layer respectively to obtain a transformed mean map corresponding to the sample mean map and a transformed feature map corresponding to the decoding feature map output by the upper decoding layer, and perform information screening on the transformed mean map based on a difference degree between the transformed mean map and the transformed feature map to obtain a target mean map.

[0140] Based on the above embodiments, the transformed mean map includes a first transformed mean map and a second transformed mean map, wherein the first transformed mean map and the second transformed mean map are obtained by performing different linear transformations on the sample mean map;

[0141] The restricted attention layer is used to generate an information threshold map based on the difference between the second transformed mean map and the transformed feature map, and to filter information from the first transformed mean map based on the information threshold map to obtain a target mean map.

[0142] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logic instructions in the memory 630 to execute a training method for a defect detection model based on dual-branch image reconstruction. This method includes: determining a normal sample image, an abnormal sample image, and a binary sample image, wherein the abnormal sample image is obtained by anomaly processing of the normal sample image, and the binary sample image reflects normal and abnormal regions in the abnormal sample image; based on an initial detection model, performing feature extraction on the abnormal sample image, and reconstructing the image based on the extracted image features to obtain a reconstructed normal image and a reconstructed abnormal image; performing defect detection based on the reconstructed normal image and the reconstructed abnormal image to obtain a defect location prediction map; and performing parameter iteration on the initial detection model based on the consistency between the defect location prediction map and the binary sample image, the reconstructed normal image and the reconstructed abnormal image in normal regions, and the consistency between the reconstructed abnormal image and the abnormal sample image in abnormal regions to obtain a defect detection model.

[0143] Moreover, the logic instructions in the memory 630 described above can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0144] In another aspect, the present application also provides a computer program product, which comprises a computer program stored on a non-transitory computer readable storage medium, and the computer program comprises program instructions, when the program instructions are executed by a computer, the computer can execute the training method of the defect detection model based on the dual-branch image reconstruction provided by the above-mentioned method, and the method comprises: determining a sample normal image, a sample abnormal image and a sample binary image, the sample abnormal image is obtained by abnormal processing based on the sample normal image, and the sample binary image is used to reflect normal regions and abnormal regions in the sample abnormal image; based on an initial detection model, performing feature extraction on the sample abnormal image, and performing image reconstruction based on the image features obtained by feature extraction to obtain a reconstructed normal image and a reconstructed abnormal image, performing defect detection based on the reconstructed normal image and the reconstructed abnormal image to obtain a defect position prediction map; based on the consistency of the defect position prediction map and the sample binary image, the reconstructed normal image and the reconstructed abnormal image on the normal region, and the consistency of the reconstructed abnormal image and the sample abnormal image on the abnormal region, performing parameter iteration on the initial detection model to obtain a defect detection model.

[0145] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements a training method of a defect detection model based on double-branch image reconstruction, the method comprising: determining a sample normal image, a sample abnormal image and a sample binary image, the sample abnormal image being obtained by performing abnormal processing on the sample normal image, and the sample binary image being used to reflect normal regions and abnormal regions in the sample abnormal image; performing feature extraction on the sample abnormal image based on an initial detection model, and performing image reconstruction based on image features obtained by the feature extraction to obtain a reconstructed normal image and a reconstructed abnormal image, and performing defect detection based on the reconstructed normal image and the reconstructed abnormal image to obtain a defect position prediction map; and performing parameter iteration on the initial detection model based on consistency of the defect position prediction map and the sample binary image, the reconstructed normal image and the reconstructed abnormal image on the normal regions, and consistency of the reconstructed abnormal image and the sample abnormal image on the abnormal regions, to obtain a defect detection model.

[0146] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0147] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software and necessary universal hardware platforms, and of course can also be implemented by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0148] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A training method for a defect detection model based on dual-branch image reconstruction, characterized in that, include: A normal sample image, an abnormal sample image, and a binary sample image are determined. The abnormal sample image is obtained by performing anomaly processing on the normal sample image. The binary sample image is used to reflect the normal and abnormal regions in the abnormal sample image. Based on the initial detection model, feature extraction is performed on the abnormal sample image, and image reconstruction is performed based on the image features obtained from the feature extraction to obtain a reconstructed normal image and a reconstructed abnormal image. Defect detection is performed based on the reconstructed normal image and the reconstructed abnormal image to obtain a defect location prediction map. Based on the difference between the defect location prediction map and the sample binary map, the consistency between the reconstructed normal image and the reconstructed abnormal image in the normal region, and the consistency between the reconstructed abnormal image and the sample abnormal image in the abnormal region, the parameters of the initial detection model are iterated to obtain the defect detection model. The initial detection model includes a reconstruction network and a segmentation network, wherein the reconstruction network includes an encoder, a first decoder, and a second decoder; The encoder is used to extract features from the abnormal sample image to obtain image features. The first decoder is used to reconstruct the image based on the sample mean map and the image features to obtain a reconstructed normal image. The sample mean map is calculated based on the pixel points of each normal sample image. The second decoder is used to reconstruct the image based on the image features to obtain a reconstructed abnormal image. The segmentation network is used to perform defect detection based on the reconstructed normal image and the reconstructed abnormal image to obtain a defect location prediction map.

2. The training method for the defect detection model based on dual-branch image reconstruction according to claim 1, characterized in that, The initial detection model is iterated based on the differences between the defect location prediction map and the sample binary map, the consistency between the reconstructed normal image and the reconstructed abnormal image in the normal region, and the consistency between the reconstructed abnormal image and the sample abnormal image in the abnormal region, to obtain a defect detection model, including: The segmentation loss is determined based on the difference between the defect location prediction map and the sample binary map; Determine the consistency between the sample normal image and the reconstructed normal image; Based on the consistency between the sample normal image and the reconstructed normal image, the consistency between the reconstructed normal image and the reconstructed abnormal image in the normal region, and the consistency between the reconstructed abnormal image and the sample abnormal image in the abnormal region, the consistency loss is determined. Based on the segmentation loss and the consistency loss, the parameters of the initial detection model are iterated to obtain the defect detection model.

3. The training method for the defect detection model based on dual-branch image reconstruction according to claim 1, characterized in that, The first decoder contains multiple decoding layers, and each decoding layer is followed by a restricted attention layer; The restricted attention layer is used to filter information based on the sample mean map and the decoded feature map output by the upper decoding layer to obtain the target mean map; The decoding layer is used to decode based on the image features, and to reconstruct the image based on the decoded feature map and the target mean map to obtain a reconstructed normal image.

4. The training method for the defect detection model based on dual-branch image reconstruction according to claim 3, characterized in that, The restricted attention layer is used to perform linear transformations on the sample mean map and the decoded feature map output by the upper decoding layer, respectively, to obtain the transformed mean map corresponding to the sample mean map and the transformed feature map corresponding to the decoded feature map output by the upper decoding layer. Based on the difference between the transformed mean map and the transformed feature map, information filtering is performed on the transformed mean map to obtain the target mean map.

5. The training method for the defect detection model based on dual-branch image reconstruction according to claim 4, characterized in that, The transformed mean plot includes a first transformed mean plot and a second transformed mean plot, which are obtained by performing different linear transformations on the sample mean plot. The restricted attention layer is used to generate an information threshold map based on the difference between the second transformed mean map and the transformed feature map, and to filter information from the first transformed mean map based on the information threshold map to obtain a target mean map.

6. A training device for a defect detection model based on dual-branch image reconstruction, characterized in that, include: A determining unit is used to determine a normal sample image, an abnormal sample image, and a binary sample image. The abnormal sample image is obtained by performing anomaly processing on the normal sample image, and the binary sample image is used to reflect the normal and abnormal regions in the abnormal sample image. A detection unit is used to extract features from the abnormal sample image based on an initial detection model, and to reconstruct the image based on the extracted features to obtain a reconstructed normal image and a reconstructed abnormal image. Defect detection is then performed based on the reconstructed normal image and the reconstructed abnormal image to obtain a defect location prediction map. The initial detection model includes a reconstruction network and a segmentation network. The reconstruction network includes an encoder, a first decoder, and a second decoder. The encoder extracts features from the abnormal sample image to obtain image features. The first decoder reconstructs the image based on the sample mean map and the image features to obtain a reconstructed normal image. The sample mean map is calculated based on the pixel points of each normal sample image. The second decoder reconstructs the image based on the image features to obtain the reconstructed abnormal image. The segmentation network performs defect detection based on the reconstructed normal image and the reconstructed abnormal image to obtain a defect location prediction map. The training unit is used to iterate the parameters of the initial detection model based on the difference between the defect location prediction map and the sample binary map, the consistency between the reconstructed normal image and the reconstructed abnormal image in the normal region, and the consistency between the reconstructed abnormal image and the sample abnormal image in the abnormal region, to obtain the defect detection model.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the training method for the defect detection model based on dual-branch image reconstruction as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method of the defect detection model based on dual-branch image reconstruction as described in any one of claims 1 to 5.

Images