A method for industrial image feature extraction based on adversarial complementary UNet
Through two-stage confrontation complementary UNet networks and fixed threshold erasing methods, the problems of low segmentation accuracy and defect misalignment in the prior art of industrial image feature extraction are solved, and higher segmentation accuracy and feature extraction accuracy are achieved.
Patent Information
- Application Number
- CN202310447603.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-24
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2043-04-24
AI Technical Summary
The existing industrial image feature extraction method based on UNet variants mainly improves the segmentation results from significant areas in the convex image, ignores the features of other areas, resulting in insufficient image feature extraction, low segmentation accuracy and defect misalignment.
Two-stage confrontation and complementary UNet networks are adopted. In the first stage, image segmentation is performed through the original UNet, fixed thresholds are set to erase uncertain areas, and the second stage is input again to train the results after erasing, and the two-stage segmentation results are fused to improve the final segmentation. Accuracy.
By countering the two-stage network and fixed threshold erasing method of complementary UNet, the image segmentation accuracy is significantly improved, the defect misalignment of segmentation results is reduced, and the accuracy of industrial image feature extraction is improved.
Smart Images

Figure CN117115458B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of image feature extraction, and in particular relates to an industrial image feature extraction method based on adversarial complementary UNet. Background Art
[0002] Machine vision has gradually penetrated into all aspects of social life. It can be seen in almost every field, including face recognition, image recognition, video surveillance, 3C applications, etc. For the industrial field, the application of machine vision has greatly reduced the risk factor of high-risk operations and ensured the safety and efficiency of industrial production.
[0003] Machine vision is very important in industrial product defect detection because it can achieve efficient, accurate and consistent automatic detection and classification; machine vision systems can automatically detect defects, eliminating the tedious and time-consuming process of manual detection, which can improve production efficiency and reduce labor costs; machine vision systems can achieve very high accuracy because they can perform more detailed and accurate detection than humans, which can reduce product rework and repeated production costs caused by detection errors; at the same time, machine vision systems can guarantee the same detection results under abnormal circumstances, whether it is day shift or night shift, or different operators; finally, machine vision systems can learn and continuously optimize algorithms and models to make defect detection and classification more accurate and intelligent, further improving the quality and efficiency of product detection.
[0004] In recent years, with the popularity of deep learning, many excellent semantic segmentation algorithms have emerged in the field of computer vision for image feature extraction. Currently, most mainstream image feature extraction models are based on UNet variants: for example, adding the residual module to the original UNet network resulted in a deep ResUNet; adding the attention gate to the decoder of UNet resulted in the Attention UNet model, which highlights the segmented target by suppressing the feature response of irrelevant background areas without introducing a large number of model parameters; adding the dilated convolution to UNet resulted in the DinkNet34 model, which expands the receptive field while maintaining resolution and won the DeepGlobe2018 Challenge, etc.
[0005] However, the current industrial image feature extraction method based on UNet variants mainly improves the segmentation results from the perspective of salient areas in the image, ignoring the features of other areas in the image, which leads to insufficient image feature extraction, insufficient final segmentation accuracy, and defective and misaligned segmentation results. Summary of the invention
[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and to provide an industrial image feature extraction method based on adversarial complementary UNet, which has the advantages of high segmentation accuracy, fine segmentation results, and high accuracy of industrial image extraction features.
[0007] To achieve the above object, the technical solution adopted by the present invention is: an industrial image feature extraction method based on adversarial complementary UNet, comprising the following steps:
[0008] S1 obtains the data set after image preprocessing;
[0009] S2 puts the data set into the original UNet for training to obtain the segmentation results of the first stage;
[0010] S3 sets a fixed threshold and erases pixels larger than the threshold in the segmentation result of the first stage;
[0011] S4 inputs the segmentation result of the first stage after pixel erasure into the new original UNet for training to obtain the segmentation result of the second stage;
[0012] S5 fuses the segmentation result of the first stage and the segmentation result of the second stage to obtain a fused result.
[0013] Furthermore, the original UNet includes but is not limited to UNet, ResUNet, Attention Unet and UNet-AcoL
[0014] Furthermore, based on the fusion results, the original UNet is trained to obtain an optimized UNet model;
[0015] Input the data set into the optimized UNet model for training, and save the network weights after the training.
[0016] Load the network weights, use the data set for testing, calculate the average intersection-over-union (mIOU), and thus obtain the most suitable optimized UNET model.
[0017] Furthermore,
[0018] Among them, p ii To predict the number of correct elements, p ij Indicates the number of true values i that are predicted to be j, p ji It indicates that the true value is j and the predicted value is i, and k indicates the number of categories to be classified.
[0019] Further, the image preprocessing method is as follows:
[0020] First, take a 512×512 bounding box template and slide it on the 1500×1500 image. First, crop the four corners and the middle image. Then, take the image between each two adjacent templates in the four bounding boxes at the four corners.
[0021] After removing some incorrectly labeled images, we finally obtained 8960 images of size 512×512. According to the division of the original dataset into training set, validation set and test set, we obtained 8361 training images, 126 validation images and 433 test images respectively.
[0022] Furthermore, the segmentation result of the first stage is P unet , assuming X H×W×C represents the image captured by the industrial camera, F unet represents the output feature map of the original UNet, P unet Represents the output feature map F of the original UNet unet The feature map obtained by the Sigmoid function, σ(·) represents the Sigmoid function, then
[0023] F unet =UNet1(X H×W×C )
[0024] P unet =σ(F unet ).
[0025] Furthermore, we set a fixed threshold value of δ and transform the feature map F unet The pixels greater than the threshold δ are erased to obtain the erased feature map F' unet , F' unet =E(F unet ,δ), where E(·) represents the fixed threshold erasing function;
[0026] The erasure function E(·) is implemented as follows:
[0027]
[0028] Among them, F unet (i,j) represents the value of the i-th row and j-th column in the output feature map of the original UNet.
[0029] Furthermore, the segmentation result of the second stage is F” unet ;P' unet Defined as the new UNet output feature map F" unet The feature map obtained by the Sigmoid function, σ(·) represents the Sigmoid function;
[0030] F” unet=UNet2(F u ' net )
[0031] P' unet =σ(F” unet ).
[0032] Furthermore, the segmentation results of the first stage and the second stage are fused to obtain the result P final ;
[0033] P final =Fusion(P unet ,P' unet ) where Fusion(·) represents the fusion function;
[0034] The fusion function is implemented as follows:
[0035]
[0036] Among them, P unet , P' unet Represents the two segmentation images to be fused, P unet (i,j),P' unet (i, j) represents the pixel value at position (i, j) of the two feature maps to be fused.
[0037] Due to the application of the above technical solution, the present invention has the following advantages compared with the prior art:
[0038] The industrial image feature extraction method based on adversarial complementary UNet of the present invention, first, the use of a two-stage network such as adversarial complementary UNet can alleviate the problem of low segmentation accuracy of a single-stage network to a certain extent; second, by fusing the two-stage segmentation results in the adversarial complementary UNet, the segmentation accuracy of the final result can be further improved; third, the use of a fixed threshold erasing method can enable the second stage of the network to learn those parts that are difficult to segment out by the original UNet in the first stage, so that the second original UNet can perform "targeted learning", thereby improving the segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] The technical solution of the present invention is further described below in conjunction with the accompanying drawings:
[0040] Figure 1 A schematic diagram of a flow chart of an embodiment of the present invention;
[0041] Figure 2 A schematic diagram of a segmentation method of an industrial image in one embodiment of the present invention;
[0042] Figure 3 This is a network structure diagram of the original UNet in one embodiment of the present invention;
[0043] Figure 4 This is an experimental comparison diagram of the segmentation results of UNet, ResUNet, Attention UNet, and UNet-AcoL in one embodiment of the present invention;
[0044] Figure 5 This is a comparison chart of the segmentation result data of UNet, ResUNet, Attention UNet, and UNet-AcoL in one embodiment of the present invention. DETAILED DESCRIPTION
[0045] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0046] The present invention provides an industrial image feature extraction method based on adversarial complementary UNet to solve the problem in the prior art that the segmentation accuracy of the extracted image is insufficient and the segmentation result has serious defects and misalignment, thus leading to inaccurate image extraction.
[0047] For ease of understanding, the specific process in the embodiments of the present application is described below. Figure 1 In an embodiment of the present application, an industrial image feature extraction method based on adversarial complementary UNet comprises the following steps:
[0048] S1, obtain the data set after image preprocessing;
[0049] In step S1, for the industrial images that have been divided into training, testing, and verification, the original 1500×1500 images (a total of 1171 images) cannot be directly trained due to their high resolution, so they need to be preprocessed and cropped to 512×512 images.
[0050] The specific method is to first take a 512×512 bounding box template, slide this template on the 1500×1500 image (base image), first crop the four corners and the middle image, and then take the image between each two adjacent templates in the four bounding boxes at the four corners in turn. For the specific segmentation method, please refer to Figure 2; Then, some incorrectly labeled images were removed, and finally 8960 images of size 512×512 were obtained. According to the division of the original data set into training set, validation set and test set, 8361 training images, 126 validation images and 433 test images were obtained respectively.
[0051] S2. Put the data set into the original UNet for training to obtain the first stage segmentation result P unet ,The dataset here refers to 8361 training images;
[0052] See also Figure 3 The original UNet network structure shown in Figure 2 assumes that X H×W×C represents the image captured by the industrial camera, F unet represents the output feature map of the original UNet, P unet Represents the output feature map F of the original UNet unet The feature map obtained by the Sigmoid function, σ(·) represents the Sigmoid function, then
[0053] F unet =UNet1(X H×W×C )
[0054] P unet =σ(F unet ).
[0055] S3, setting a fixed threshold, erasing pixels greater than the threshold in the segmentation result of the first stage;
[0056] In step S3, a fixed threshold δ is set to transform the feature map F unet The pixels greater than the threshold δ are erased to obtain the erased feature map F' unet , E(·) represents the fixed threshold erasing function;
[0057] F' unet =E(F unet ,δ)
[0058] The specific implementation method of the erasure function E(·) is as follows:
[0059]
[0060] Specifically, the present invention uses δ to represent the threshold, F unet (i, j) represents the value of the i-th row and j-th column in the output feature map of the original UNet. Using threshold erasing can not only erase the wrong segmentation area, but also enable the multi-scale void convolution UNet to segment the defective area that is more difficult to segment.
[0061] S4, erase the segmentation result F u 'net Input into the new original UNet for training to obtain the second stage segmentation result F" unet , that is, using another same original UNet to erase the segmentation result F' unet Conduct training;
[0062] In step S4, P' unet Defined as the new original UNet output feature map F' unet The feature map obtained by the Sigmoid function; σ(·) represents the Sigmoid function;
[0063] F” unet =UNet2(F' unet )
[0064] P' unet =σ(F” unet )
[0065] S5, the segmentation result P of the first stage unet And the segmentation result P' of the second stage unet Fusion, the fusion result P final ;
[0066] In step S5, P final As the final segmentation result of the adversarial complementary UNet, Fusion(·) represents the fusion function;
[0067] P final =Fusion(P unet ,P' unet )
[0068] The implementation details of the fusion function are as follows:
[0069]
[0070] Among them, P unet , P' unet Represents the two segmentation images to be fused, P unet (i,j),P' unet (i, j) represents the pixel value at position (i, j) of the two feature maps to be fused.
[0071] The above formula indicates that when the first-stage segmentation map and the second-stage segmentation map are obtained, the regions that are not segmented in the two feature maps are complementary fused to further improve the segmentation accuracy of the final model.
[0072] In the present invention, in order to find the optimized Unet model that is most suitable for application and industry, and to facilitate subsequent direct use, the following steps are added:
[0073] S6. Based on the fusion results, the original UNet is trained to obtain an optimized UNet model;
[0074] S7, input the data set into the optimized UNet model for training, and save the network weights after the training is completed;
[0075] S8. Load the network weights, use the data set for testing, calculate the average intersection-over-union (mIOU), and thus obtain the most suitable optimized UNet model.
[0076] The following are the methods for loading network weights, testing with the test set, and calculating the average intersection-over-union ratio and mDC:
[0077]
[0078] Among them, p ii To predict the number of correct elements, p ij Indicates the number of true values i that are predicted to be j, p ji It indicates that the true value is j and the predicted value is i, and k indicates the number of categories to be classified.
[0079] See also Figure 5 ,From the figure, we can see that among the finally calculated ,intersection over union ratio mIOU, UNet-ACoL has the highest accuracy, which means that the segmentation result is more accurate ,image extraction has the highest accuracy.
[0080] In addition, the segmentation results of UNet, ResUNet, Attention UNet and Adversarial Complementary UNet (UNet-ACoL) are compared. Figure 4 shown.
[0081] This paper applies the adversarial complementary UNet to the task of industrial image feature extraction for the first time. In the defect detection of industrial images, the segmentation accuracy far exceeds that of traditional single-stage networks such as UNet, ResUNet, and Attention UNet, which has great practical significance for the development of industrial production automation lines.
[0082] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. An industrial image feature extraction method based on adversarial complementary UNet, It is characterized in that The steps include: S1 obtains the data set after image preprocessing; S2 puts the data set into the original Unet for training to obtain the segmentation results of the first stage; S3 sets a fixed threshold and erases pixels larger than the threshold in the segmentation result of the first stage; S4 inputs the segmentation result of the first stage after pixel erasure into the new original Unet for training to obtain the segmentation result of the second stage; S5 fuses the segmentation result of the first stage and the segmentation result of the second stage to obtain a fused result; The segmentation result of the first stage is P unet , assuming X H×W×C represents the image captured by the industrial camera, F unet represents the output feature map of the original UNet, P unet Represents the output feature map F of the original UNet unet The feature map obtained by the Sigmoid function, σ(·) represents the Sigmoid function, then F unet =UNet 1 (X H×W×C ) P unet =σ(F unet ); Set the fixed threshold to δ and transform the feature map F unet The pixels greater than the threshold δ are erased to obtain the erased feature map F u ' net , F u ' net =E(F unet ,δ), where E(·) represents the fixed threshold erasing function; The erasure function E(·) is implemented as follows: Among them, F unet (i,j) represents the value of the i-th row and j-th column in the output feature map of the original UNet; the segmentation result of the second stage is F″ unet ; P′ unet Defined as the new UNet output feature map F″ unet The feature map obtained by the Sigmoid function, σ(·) represents the Sigmoid function; F″ unet =UNet 2 (F′ unet ) P′ unet =σ(F″ unet ); The segmentation results of the first stage and the second stage are fused to obtain the result P final ; P final =Fusion(P unet ,P′ unet ) where Fusion(·) represents the fusion function; The fusion function is implemented as follows: Among them, P unet , P u ' net Represents the two segmentation images to be fused, P unet (i,j),P u ' net (i, j) represents the pixel value at position (i, j) of the two feature maps to be fused.
2. The industrial image feature extraction method based on adversarial complementary UNet as claimed in claim 1, Features: Based on the fusion results, the original UNet is trained to obtain the optimized UNet model; Input the data set into the optimized UNet model for training, and save the network weights after the training. Load the network weights, use the data set for testing, calculate the average intersection-over-union (mIOU), and thus obtain the most suitable optimized UNet model.
3. The industrial image feature extraction method based on adversarial complementary UNet as claimed in claim 2, Features: Among them, p ii To predict the number of correct elements, p ij Indicates the number of true values i that are predicted to be j, p ji It indicates that the true value is j and the predicted value is i, and k indicates the number of categories to be classified.
4. The industrial image feature extraction method based on adversarial complementary UNet as claimed in claim 1, It is characterized in that The image preprocessing method is as follows: First, take a 512×512 bounding box template and slide it on the 1500×1500 image. First, crop the four corners and the middle image. Then, take the image between each two adjacent templates in the four bounding boxes at the four corners. After removing some incorrectly labeled images, we finally obtained 8960 images of size 512×512. According to the division of the original dataset into training set, validation set and test set, we obtained 8361 training images, 126 validation images and 433 test images respectively.
