A bolt anomaly detection method in complex scenes based on deep feature reconstruction
By using deep feature reconstruction technology in complex scenarios, deep features are extracted and fused, bolt position prediction and abnormal detection are realized, and the problem of poor bolt abnormal detection effect in complex scenarios is solved, and efficient and accurate abnormal detection and feature reconstruction are achieved.
Patent Information
- Application Number
- CN202210208939.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-04
- Publication Date
- 2025-03-25
- Estimated Expiration
- 2042-03-04
AI Technical Summary
The existing abnormality detection technology is difficult to effectively detect bolt abnormalities in complex scenarios, especially in high-speed rail grease injection equipment with large changes in lighting conditions and background. The existing methods cannot effectively locate bolts and reduce the effect of abnormality detection.
The deep feature reconstruction method is adopted to extract deep features through convolutional downsampling network, and feature fusion and reconstruction are performed in convolutional interpolation upsampling network to achieve prediction and abnormal detection of bolt position, and generate the outline of bolts in normal state.
The accuracy and efficiency of bolt abnormality detection are improved in complex scenarios, and the positioning, abnormality detection and feature reconstruction of key components can be achieved simultaneously, which is suitable for supervision tasks in actual production scenarios.
Smart Images
Figure CN114897765B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of machine vision detection, and in particular to a method for detecting bolt anomalies in complex scenes based on deep feature reconstruction. Background Art
[0002] Anomaly detection is a popular task in the field of computer vision. It is widely used in the field of industrial production to liberate manpower, improve productivity and production efficiency, and has high use value. Its specific application scenarios include product quality inspection, production scene supervision, etc. Due to the wide range of application scenarios, the methods used require a high level of scene understanding ability. At the same time, due to the unpredictability of anomalies, it is impossible to predict all possible anomalies and the number of abnormal samples is small. Therefore, it can only use non-abnormal samples to extract prior knowledge, that is, semi-supervised or unsupervised methods, which brings great difficulty to anomaly detection tasks. At the same time, this is also the main feature that distinguishes anomaly detection tasks from defect detection tasks.
[0003] The existing anomaly detection technologies are mainly the following:
[0004] The first category is methods based on deep features: this type of method uses a deep neural network to extract the deep features of samples without anomalies. The deep features of these samples without anomalies can be directly stored in a database as prior knowledge of no anomalies, or the distribution pattern of the deep features of samples without anomalies can be extracted. When performing anomaly detection, the same deep network is used for feature extraction, and the error is calculated by comparing with the prior knowledge of no anomalies. When the error is large, it can be determined that an anomaly exists.
[0005] The second category is based on feature reconstruction methods: This type of method first uses an encoder to encode the input image, extracts its core features, and uses its encoding result as input and the original non-anomaly image as output to train a reconstruction network. Since all training images are non-anomaly samples, the reconstruction network only has the ability to reconstruct non-anomaly images. When performing anomaly detection, the input image and its reconstructed image are compared, and places with large differences can be judged as anomalies.
[0006] However, both of these methods are only applicable to relatively regular scenes, such as product quality inspection. However, in the application scenario of the present invention, not only the lighting conditions and background change greatly, but also the high-speed rail grease injection port equipment (as shown in Figure (1a)) itself has great differences. The change of the parking position of the train also leads to the change of the perspective of the picture, which leads to the change of the position of the equipment in the picture. Figure (1b) shows some samples without abnormalities and marks the bolts. The change of perspective and scene not only makes it impossible for the existing methods to focus on key components such as bolts, making it difficult to summarize the prior knowledge of detecting abnormalities, but also reduces the effect of the existing methods in abnormality detection. Summary of the invention
[0007] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a method for bolt anomaly detection in complex scenes based on deep feature reconstruction. For this specific scene, the present invention combines the anomaly detection task with the positioning of key components such as bolts. At the same time, in order to obtain a higher detection speed, the network proposed by this method can complete multiple tasks at the same time, including positioning of key components, anomaly detection and reconstruction of the outline of the bolt in a normal state.
[0008] The purpose of the present invention can be achieved by the following technical solutions:
[0009] A method for detecting bolt anomalies in complex scenes based on deep feature reconstruction, the method comprising the following steps:
[0010] 1) Obtain the original image of the mechanical equipment in the complex scene, input it into the convolutional downsampling network, and obtain the deep feature maps of different resolutions to retain the structural information of the mechanical equipment and the visual information of the bolts;
[0011] 2) The depth feature maps of different resolutions are fused and input into the convolutional interpolation upsampling network, the depth features are decoded and the input feature map is restored to the original image size, and a matrix containing the unit vector of each pixel in the image predicted to the bolt and a mask map of a normal bolt, i.e., the original mask map, is generated. The position of each bolt is obtained by predicting the matrix containing the unit vector of each pixel in the image predicted to the bolt and combining it with the position of each pixel itself;
[0012] 3) According to the obtained bolt position, the features near the bolt position are removed from the feature maps of different resolutions obtained by the convolutional downsampling network, and the features of the bolt area in a normal state are reconstructed, so as to complete the feature map, and perform bolt detection on the reconstructed feature map to obtain a mask map of all bolts of the mechanical equipment in a normal state, i.e., a reconstructed mask map;
[0013] 4) Compare the original mask image and the reconstructed mask image to complete the determination of abnormal bolts and generate the final abnormal bolt restoration image.
[0014] In the step 1), the convolutional downsampling network has four layers, of which the last three layers adopt deep residual convolutional networks, and batch normalization and ReLU activation function are performed after each convolution layer.
[0015] In the step 2), the depth feature maps of different resolutions obtained by the convolutional downsampling network are input into the convolutional interpolation upsampling network after feature splicing and fusion through jump connections, wherein the depth feature map extracted by the last layer in the convolutional downsampling network is directly used as the input of the convolutional interpolation upsampling network, and the feature maps extracted by the first three layers in the convolutional downsampling network are added by splicing during the convolutional upsampling process to provide feature information of different levels for the convolutional interpolation upsampling network.
[0016] The feature splicing and fusion method is specifically as follows:
[0017] The two features are combined in the channel dimension to obtain a thicker feature.
[0018] The convolutional interpolation upsampling network includes 5 standard convolutional layers and 4 bilinear interpolation upsampling layers. The first four standard convolutional layers are followed by a bilinear interpolation upsampling layer to restore the scale of the feature map. Each bilinear interpolation upsampling layer is spliced with the depth feature map of the corresponding scale in the convolutional downsampling network to obtain features with semantic information at different levels, and aggregate contextual semantic information based on receptive fields of different regions and sizes. The last standard convolutional layer serves as channel compression.
[0019] The convolutional interpolation upsampling network uses a bilinear interpolation method to expand the feature image and restore the input feature map to the original image size.
[0020] The convolutional interpolation upsampling network outputs the original mask image through an instance segmentation head, and outputs a matrix of unit vectors predicting each pixel in the image to the bolt through a position voting head. The instance segmentation head has a fully connected structure.
[0021] In the step 3), according to the obtained bolt position, the features near the bolt position are removed from the feature maps of different resolutions obtained by the convolutional downsampling network, specifically:
[0022] Draw a circle with each predicted bolt position as the center, and determine the radius of the circle according to the correspondence between the feature map scale and the original image scale, so that the circle covers the corresponding bolt, and set the feature values of the parts within the circle in the feature maps of different resolutions to zero.
[0023] The step 4) is specifically as follows:
[0024] If a bolt is not detected in the original mask image but is detected in the reconstructed mask image, it is determined to be abnormal, and the result in the reconstructed mask image is used as the outline of the bolt in a normal state.
[0025] The neural network used for instance segmentation and reconstruction features uses only non-abnormal samples during the training phase, so the original mask image contains the contours of the bolts in normal state in the original image, and the reconstructed mask image contains the contours of all bolts in normal state.
[0026] Compared with the prior art, the present invention has the following advantages:
[0027] 1. The present invention combines the position prediction method with the anomaly detection method, so that the method can solve the anomaly detection problem in complex scenes with background changes, perspective changes and light changes, and only uses non-abnormal samples for training, so it is easy to deploy in actual production scenarios.
[0028] 2. The present invention designs a deep neural network that can simultaneously realize the functions of position prediction, anomaly detection and reconstruction of abnormal key components, thereby reducing the lag in the feature extraction process and improving the efficiency of anomaly detection.
[0029] 3. The present invention only reconstructs the features near the key components, so that a neural network with a relatively light structure can be used to implement the above functions, eliminating the reconstruction process of invalid features and improving the efficiency of feature reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 This is a real scene diagram of the application of the present invention (equipment at the grease injection port of a high-speed train), where Figure (1a) is an unlabeled original image and Figure (1b) is an example of key component labeling.
[0031] Figure 2 It is the model prediction result in the embodiment, wherein Figure (2a) is the predicted position of the key component, Figure (2b) is the original semantic segmentation result, Figure (2c) is the reconstructed semantic segmentation result, Figure (2d) is the generated semantic segmentation map of the abnormal component, and Figure (2e) is the result of the semantic segmentation map of the abnormal component on the original image.
[0032] Figure 3 This is a structural diagram of the deep neural network of the present invention. In the figure, the arrow indicates the direction of feature transmission, the black solid line indicates the original feature flow, and the gray dotted line indicates the reconstructed feature flow.
[0033] Figure 4 Schematic diagram of the structure of the Conv3x3-BN-ReLU module (ordinary two-dimensional convolution module).
[0034] Figure 5 Schematic diagram of the structure of the local feature reconstruction module.
[0035] Figure 6 To combine the semantic segmentation map of each key component into a semantic segmentation result map. DETAILED DESCRIPTION
[0036] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work should belong to the scope of protection of the present invention.
[0037] Example
[0038] The present invention provides a bolt anomaly detection method based on deep feature reconstruction. The method improves the computational efficiency by simultaneously realizing key point positioning and semantic segmentation of key components. The method realizes the restoration of abnormal components through regional feature reconstruction. The method can be applied to scene supervision in industrial and production scenes. The method is described by taking a high-speed train grease injection port device in an actual scene as an example. The method specifically comprises the following steps:
[0039] S1. Obtain training images and perform semantic annotation, such as Figure 1 As shown;
[0040] S2. Build an anomaly detection network and reduce the image resolution to 1 / 16 through 4 times of learnable convolution downsampling, retaining the spatial position and visual feature information of the object while reducing information redundancy in the image;
[0041] Among them, the learnable convolution downsampling module has four layers, and the last three layers are more efficient deep residual convolution blocks. The three kernel sizes are all 3×3, and the stride is 2. Therefore, the output feature map of this module is downsampled to 1 / 16 of the original image, and after each convolution, it is batch normalized and ReLU activated.
[0042] During the downsampling process, the features of each convolutional layer output will be temporarily retained, and the features of the last layer will be directly used as the input of the convolutional interpolation upsampling network. The features of the first three layers will be added by splicing during the convolutional upsampling process, providing feature information of different levels for the convolutional interpolation upsampling network.
[0043] S3, the convolutional interpolation upsampling network uses 5 groups of convolutional layers and 4 bilinear upsampling layers to perform deep semantic feature extraction on the 1 / 16 image of the convolution downsampling. In this process, the feature maps of the first 3 levels of the downsampling network are merged into it. The convolutional interpolation upsampling network will restore the features and obtain the semantic segmentation results of the bolt contour (original mask map) and the prediction results of the bolt position.
[0044] Specifically, the size of the feature maps of the four layers of the convolutional downsampling network is 1 / 2, 1 / 4, 1 / 8, and 1 / 16 of the input image, and the number of channels is 64, 128, 256, and 512, respectively. The lower-level features mainly include the edge, color, and other contour information of the input image, and the higher-level features will contain information that requires semantic understanding, such as different components of the device to be detected. Due to the merging of features, the number of channels of the input features of the first four convolutional layers of the convolutional interpolation upsampling network are 512, 512, 256, and 128, respectively. The first three convolutional layers will be followed by a bilinear upsampling layer to restore the scale of the feature map. The last convolutional layer uses a 1x1 convolution kernel, which is used to compress the channels of the features.
[0045] The upsampling modules in the network all use bilinear interpolation methods, which can efficiently expand the feature image so that the final output image has the same resolution as the input image.
[0046] S4. Using the prediction results of the position of each key component output by the convolution interpolation upsampling network, the features of each key component in the four levels of features in the convolution downsampling network are removed.
[0047] Specifically, a circle is drawn with the predicted position of each key component as the center, and the radius of the circle is determined according to the correspondence between the feature map scale and the original image scale, so that the circle should cover the corresponding key component, and the value of the part of the feature map within the circle is set to zero.
[0048] Each pixel in the image will predict the vector of the bolt position relative to its own position based on its own characteristics and the structural information of the production equipment, thereby generating a matrix containing the prediction results of each pixel, which is used to calculate the final predicted bolt position.
[0049] S5. Use the result of the previous step as the input of the regional feature reconstruction module to obtain the feature map after reconstructing the features of the area where the key component is located, and input it into the convolutional interpolation upsampling network to obtain the reconstructed semantic segmentation result of the key component.
[0050] Specifically, since the convolutional downsampling network outputs features at four different levels, the regional feature reconstruction module will also be composed of four parts, and the reconstructed features at four different levels are input into the convolutional interpolation upsampling network.
[0051] S6. Compare the original semantic segmentation result and the reconstructed semantic segmentation result to generate the final restoration image of the abnormal part.
[0052] Specifically, key components that exist in the reconstructed semantic segmentation result but not in the original semantic segmentation result are judged to be abnormal, and the semantic segmentation results of these abnormal components in the reconstructed segmentation result are their positions when they are in a normal state.
[0053] Figure 3 The names and functions of each module are explained as follows:
[0054]
[0055] The specific implementation process of the method of the present invention is given below:
[0056] Step S1: Collect sample images without abnormalities and perform semantic annotation. Figure 1 shown.
[0057] Step S2: Build a semantic segmentation network according to the layout of each layer of the semantic segmentation network shown in Table 1. As shown in Table 1, the number of channels of the four learning convolution downsamplings is 64, 128, 256, and 512, the convolution kernel size is 3×3, the step size is 2, and the size of the output feature map is 256×256, 128×128, 64×64, and 32×32, respectively, as shown in Table 1. The last three are deep residual convolution layers, which contain multiple convolution layers that can keep the feature size unchanged, thereby extracting feature maps with deeper semantic understanding; the feature maps of different levels of the convolution downsampling output can be used to extract deep semantic information, as well as extract shallow contour boundary information of the image.
[0058] Table 1 Layout of each layer of the convolutional downsampling network
[0059]
[0060]
[0061] Step S3: Figure 3 As shown in Table 2, the convolution upsampling layer consists of 5 standard convolution layers and 4 bilinear interpolation upsampling layers. The convolution kernel size of the first four convolution layers is 3x3, followed by a bilinear interpolation upsampling layer to restore the scale of the feature map. Each bilinear upsampling layer restores the scale of the feature map and concatenates it with the feature map of the corresponding scale in the convolution downsampling network, thereby obtaining features with different levels of semantic information and aggregating contextual semantic information based on different regions and receptive fields of different sizes. The last convolution layer uses a 1x1 convolution kernel, which acts as channel compression. The number of channels in the final output feature map is 3*11, where 11 is the number of key components, representing the x and y coordinates of the position of each key component, as shown in Figure (2a), and the original semantic segmentation result of the key component (original mask map), as shown in Figure (2b).
[0062] Table 1 Layout of each layer of the convolution interpolation upsampling network
[0063] enter Modules Number of channels Step length Output 32×32×512 Conv2D 256 1 32×32×256 32×32×256 Bilinear layer 64×64×256 64×64×512 Conv2D 128 1 64×64×128 64×64×128 Bilinear 128×128×128 128×128×256 Conv2D 64 1 128×128×64 128×128×64 Bilinear 256×256×64 256×256×128 Conv2D 32 1 256×256×32 256×256×32 Bilinear 512×512×32 512×512×32 Conv2D 11×3 1 512×512×33
[0064] Step S4: Using the prediction results of the position of each key component output by the convolutional interpolation upsampling network, the features of each key component in the features of the four levels in the convolutional downsampling network are removed.
[0065] Specifically, a circle is drawn with the predicted position of each key component as the center, and the radius of the circle is determined according to the correspondence between the feature map scale and the original image scale, so that the circle should cover its corresponding key component. For example, if the diameter of a bolt in the original image with a scale of 512x512 is 30, then in the feature map with a scale of 256x256, the diameter of the circular area must be 15, and the feature value of the part of the feature map within the circle is set to 0.
[0066] Step S5: Use the result of the previous step as the input of the regional feature reconstruction module to obtain the feature map after reconstructing the features of the region where the key component is located, and input it into the convolutional interpolation upsampling network to obtain the reconstructed semantic segmentation result of the key component (reconstructed mask map), as shown in Figure (2c).
[0067] Step S6:
[0068] By comparing the original semantic segmentation results with the reconstructed semantic segmentation results, the key components that exist in the reconstructed semantic segmentation results (reconstructed mask map) but not in the original semantic segmentation results (original mask map) will be judged as abnormal. The semantic segmentation results of these abnormal components in the reconstructed segmentation results will be extracted to generate the final abnormal component reconstruction map, as shown in Figure (2d). The effect of its effect on the original image is shown in Figure (2e). At the same time, the positioning of key components such as bolts, abnormality detection, and restoration of abnormal components are realized.
[0069] In summary, the real-time image semantic segmentation method provided by the present invention has three major innovations compared with the methods in the prior art:
[0070] First, the present invention combines the position prediction method with the anomaly detection method, so that the method can solve the anomaly detection problem in complex scenes with background changes, perspective changes, and light changes, and only uses no samples for training, so it is easy to deploy in actual production scenarios.
[0071] Second, the present invention designs a deep neural network that can simultaneously realize the functions of position prediction, anomaly detection, and reconstruction of abnormal key components, thereby reducing the lag in the feature extraction process and improving the efficiency of anomaly detection.
[0072] Third, the present invention only reconstructs the features near the key components, so that a neural network with a relatively light structure can be used to implement the above functions, avoiding the reconstruction process of invalid features and improving the efficiency of feature reconstruction.
[0073] The above three innovations further improve the real-time performance of the anomaly detection network and improve its accuracy by predicting the positions of key components. They have achieved good results in terms of speed and accuracy. It is a real-time anomaly detection method that can be easily deployed in actual production equipment supervision scenarios.
[0074] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed by the present invention, and these modifications or substitutions should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A bolt anomaly detection method in complex scenes based on deep feature reconstruction, characterized in that: The method comprises the following steps: 1) Obtain the original image of the mechanical equipment in the complex scene, input it into the convolutional downsampling network, and obtain the deep feature maps of different resolutions to retain the structural information of the mechanical equipment and the visual information of the bolts; 2) The depth feature maps of different resolutions are fused and input into the convolutional interpolation upsampling network, the depth features are decoded and the input feature map is restored to the original image size, and a matrix containing the unit vector of each pixel in the image predicted to the bolt and a mask map of a normal bolt, i.e., the original mask map, is generated. The position of each bolt is obtained by predicting the matrix containing the unit vector of each pixel in the image predicted to the bolt and combining it with the position of each pixel itself; 3) According to the obtained bolt position, the features near the bolt position are removed from the feature maps of different resolutions obtained by the convolutional downsampling network, and the features of the bolt area in a normal state are reconstructed, so as to complete the feature map, and perform bolt detection on the reconstructed feature map to obtain a mask map of all bolts of the mechanical equipment in a normal state, i.e., a reconstructed mask map; 4) Compare the original mask image and the reconstructed mask image to complete the determination of abnormal bolts and generate the final abnormal bolt restoration image.
2. According to the method of claim 1, the method is characterized in that: In the step 1), the convolutional downsampling network has four layers, of which the last three layers adopt deep residual convolutional networks, and batch normalization and ReLU activation function are performed after each convolution layer.
3. According to the method of bolt anomaly detection in complex scenes based on deep feature reconstruction according to claim 2, it is characterized in that: In the step 2), the depth feature maps of different resolutions obtained by the convolutional downsampling network are input into the convolutional interpolation upsampling network after feature splicing and fusion through jump connections, wherein the depth feature map extracted by the last layer in the convolutional downsampling network is directly used as the input of the convolutional interpolation upsampling network, and the feature maps extracted by the first three layers in the convolutional downsampling network are added by splicing during the convolutional upsampling process to provide feature information of different levels for the convolutional interpolation upsampling network.
4. The method for detecting bolt anomalies in complex scenes based on deep feature reconstruction according to claim 3 is characterized in that: The feature splicing and fusion method is specifically as follows: The two features are combined in the channel dimension to obtain a thicker feature.
5. According to the method of bolt anomaly detection in complex scenes based on deep feature reconstruction according to claim 3, it is characterized in that: The convolutional interpolation upsampling network includes 5 standard convolutional layers and 4 bilinear interpolation upsampling layers. The first four standard convolutional layers are followed by a bilinear interpolation upsampling layer to restore the scale of the feature map. Each bilinear interpolation upsampling layer is spliced with the depth feature map of the corresponding scale in the convolutional downsampling network to obtain features with semantic information at different levels, and aggregate contextual semantic information based on receptive fields of different regions and sizes. The last standard convolutional layer serves as channel compression.
6. The method for detecting bolt anomalies in complex scenes based on deep feature reconstruction according to claim 5 is characterized in that: The convolutional interpolation upsampling network uses a bilinear interpolation method to expand the feature image and restore the input feature map to the original image size.
7. The method for detecting bolt anomalies in complex scenes based on deep feature reconstruction according to claim 5 is characterized in that: The convolutional interpolation upsampling network outputs the original mask image through an instance segmentation head, and outputs a matrix of unit vectors predicting each pixel in the image to the bolt through a position voting head. The instance segmentation head has a fully connected structure.
8. The method for detecting bolt anomalies in complex scenes based on deep feature reconstruction according to claim 1 is characterized in that: In the step 3), according to the obtained bolt position, the features near the bolt position are removed from the feature maps of different resolutions obtained by the convolutional downsampling network, specifically: Draw a circle with each predicted bolt position as the center, and determine the radius of the circle according to the correspondence between the feature map scale and the original image scale, so that the circle covers the corresponding bolt, and set the feature values of the parts within the circle in the feature maps of different resolutions to zero.
9. The method for detecting bolt anomalies in complex scenes based on deep feature reconstruction according to claim 1 is characterized in that: The step 4) is specifically as follows: If a bolt is not detected in the original mask image but is detected in the reconstructed mask image, it is determined to be abnormal, and the result in the reconstructed mask image is used as the outline of the bolt in a normal state.
10. The method for detecting bolt anomalies in complex scenes based on deep feature reconstruction according to claim 1 is characterized in that: The original mask image includes the outline images of the bolts in the original image that are in a normal state, and the reconstructed mask image includes the outline images of all the bolts in a normal state.