Open environment positive sample anomaly detection method based on thermodynamic diagram and target detection, electronic equipment and storage medium

Abstract

The invention relates to the technical field of deep learning, and provides an open environment positive sample anomaly detection method based on a thermodynamic diagram and target detection, electronic equipment and a storage medium. The objective of the invention is to solve the problems of high false detection rate and poor detection stability caused by environmental noise interference of a positive sample anomaly detection algorithm in an open environment. According to the main scheme, the method comprises the following steps: collecting a same-point multi-temporal image, selecting one test image, and taking the rest as a base image; two data sets are generated after the key points are corrected, wherein the set A contains an annotation abnormal area which is manually added, and the set B retains an original image. And a twin network is used to finely adjust the feature extractor to reduce the environmental noise difference. And extracting semantic features of the test diagram and the base diagram, and calculating a minimum mahalanobis distance to generate a thermodynamic diagram. A double-backbone network is constructed, a first backbone processes difference features of a test diagram and a base diagram, a second backbone fuses thermodynamic diagram features, and anomaly positioning classification is realized through a detection head after multi-level feature fusion. And carrying out network training by using the thermodynamic diagram generated by the data set A and the annotation information.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and provides a method for detecting anomalies of positive samples in an open environment based on heat maps and target detection, an electronic device, and a storage medium. Background Art

[0002] 1. Current status of the technology field

[0003] Positive anomaly detection algorithms, a key branch of deep learning, are widely used in scenarios such as industrial quality inspection and security monitoring. Traditional methods, such as the One-Class Support Vector Machine (SVM), distinguish between normal and abnormal samples by constructing a hyperplane. However, these methods rely on a static feature space and struggle to adapt to dynamic noise interference such as lighting and weather in open environments. In recent years, large-model-based detection technologies have improved generalization capabilities through open vocabulary recognition, but they rely on massive amounts of labeled data, and the scarcity of negative samples continues to hinder their practical deployment.

[0004] 2. Existing technical bottlenecks

[0005] Currently, positive sample anomaly detection algorithms perform poorly in open environments, and detection results fluctuate with environmental changes. The reasons for this fluctuation are:

[0006] 1. When constructing the positive sample basemap, it is impossible to cover all changes in the environment. The changes contained in the basemap are only a small subset of all changes.

[0007] 2. Anomalies in open environments cannot be enumerated, and the noise generated by environmental changes cannot be enumerated in the spatial and temporal dimensions. Therefore, it is impossible to use a supervised approach to tell the positive sample algorithm what are true anomalies and what are false positives caused by environmental interference.

[0008] In summary, the current positive sample anomaly detection algorithm's detection effect strongly depends on the degree to which the basemap contains environmental change factors. However, constructing a basemap that can cover all changes is unrealistic from the perspectives of basemap collection, basemap storage, and basemap comparison efficiency, and it is impossible to achieve a perfect basemap construction. Summary of the Invention

[0009] The present invention aims to solve the problems of high false detection rate and poor detection stability of positive sample anomaly detection algorithms in open environments due to environmental noise interference (such as illumination and weather changes). Through a dual-modal feature fusion mechanism guided by a heat map, accurate positioning of abnormal areas and effective suppression of interference noise can be achieved in a limited base map set that does not need to cover all environmental changes.

[0010] The present invention provides a method for detecting anomalies of positive samples in an open environment based on heat maps and target detection, comprising the following steps:

[0011] Step 1. Basic training dataset construction:

[0012] 1.a. Collect N images of the same point at different times, select one as the test image, and the remaining N-1 images as the base image;

[0013] 1.b. Detect and correct key points on the test image and base image to generate a corrected dataset;

[0014] 1.c. Copy the corrected dataset to generate dataset A and dataset B, where:

[0015] Dataset A adds anomaly areas to the test image and marks the outer bounding rectangle of the anomaly areas, or generates the same anomaly in the same area of ​​the test image and the base image;

[0016] Dataset B keeps the original images without adding abnormalities;

[0017] Step 2. Generate heat map of suspicious areas:

[0018] 2.a Use the twin network structure to fine-tune the feature extraction network to reduce the difference in features extracted from images with different environmental noise at the same point;

[0019] 2.b. Extract semantic features from the test image and base image, calculate the minimum Mahalanobis distance between the semantic features of the test image and each base image, and generate a heat map;

[0020] Step 3. Anomaly detection network training and detection:

[0021] 3.a. Build a dual-backbone object detection network, with the first backbone network processing the difference features between the test image and the base image, and the second backbone network processing the heat map features;

[0022] 3.b. Fusion of differential features and heat map features at multiple levels, input into the detection head for anomaly location and classification;

[0023] 3.c. Use the heatmap generated by dataset A and the anomalies to train the anomaly detection network.

[0024] In the above method, the key point detection and correction in step 1.b includes:

[0025] The SuperPoint algorithm is used to detect the key points of the test image and the base image. The key point pairs are matched using the SuperGlue algorithm. The correction transformation matrix is ​​calculated based on the matching results, and the base image is affine transformed to align it with the test image.

[0026] In the above method, the generation of the dataset A includes at least one of the following situations:

[0027] Only random abnormal areas are generated on the test image and marked as abnormalities to be detected;

[0028] The same abnormal area is generated at the same position of the test image and the base image, and marked as an interference anomaly that does not need to be detected.

[0029] In the above method, the fine-tuning of the feature extraction network in step 2.a includes:

[0030] ResNet50 is used as the backbone of the twin network, and the base map of dataset B is used as the training set. The network parameters are adjusted by minimizing the feature differences of images under different interference conditions of the same scene.

[0031] In the above method, the method of generating a heat map in step 2.b includes:

[0032] The Mahalanobis distance between the semantic features of the test image and each base image is calculated pixel by pixel, and the minimum distance value of the corresponding position of each base image is taken as the pixel value of the heat map. The high-value area in the heat map indicates potential anomalies.

[0033] In the above method, the dual-backbone object detection network in step 3.a is based on the improved YOLOv5 architecture and includes:

[0034] Copy the YOLOv5 backbone network into the first backbone network and the second backbone network;

[0035] The first backbone network extracts the difference features between the test image and the base image;

[0036] The second backbone network extracts heat map features and performs feature addition and fusion with difference features at multiple levels.

[0037] In the above method, obtaining the difference feature includes:

[0038] The test image and the base image are input into the first backbone network respectively to obtain multi-layer semantic features, and the corresponding layer features are subtracted to generate difference features A, B, and C.

[0039] In the above method, the construction of the anomaly detection training set in step 3.c includes:

[0040] A heatmap set is generated for the test image and base map in dataset A, and the test image, base map, heatmap and anomaly annotations are combined into a training set for end-to-end training of the anomaly detection network.

[0041] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program, and the processor implements the steps of the method when executing the program.

[0042] The present invention also provides a computer-readable storage medium storing computer instructions, which implement the steps of the method when the instructions are executed by a processor.

[0043] Because the present invention adopts the above technical means, it has the following beneficial effects:

[0044] 1. This invention solves the technical problem of image misalignment and false detection caused by shooting angle offset in open environments by using the technical means of SuperPoint+SuperGlue key point detection and affine transformation correction. It achieves the technical effect of eliminating the interference of perspective differences on anomaly detection through sub-pixel image correction, and reducing the pixel-level comparison error between the base image and the test image to a negligible level.

[0045] 2. The present invention solves the technical problem that traditional methods cannot distinguish between real anomalies and similar changes caused by environmental interference by constructing a data set A containing interference anomalies (the same anomalies are generated at the same position in the test image and the base image), and achieves the technical effect of training the network to automatically ignore repetitive interference patterns.

[0046] 3. The present invention solves the technical problem of feature space distribution offset caused by environmental noise such as lighting and weather through the technical means of the twin network fine-tuning strategy (minimizing the feature differences of different interference images in the same scene), and achieves the technical effect of reducing the sensitivity of the ResNet50 feature extractor to noise in the same scene.

[0047] 4. This invention solves the technical problem that a single base map comparison cannot cover multi-dimensional environmental changes by using the minimum Mahalanobis distance heat map generation algorithm. By dynamically comparing multiple base maps to locate abnormal areas, it achieves coverage of more than 90% of the true abnormal areas with a limited base map set (N=5).

[0048] 5. This invention solves the technical problems of the dual-backbone network multi-level feature fusion architecture through the dual-backbone network multi-level feature fusion architecture. The complementary fusion of difference features and heat map features enables the anomaly positioning accuracy of industrial quality inspection scenarios to reach 95%.

[0049] 6. This invention solves the technical problem of model overfitting caused by the fuzzy definition of open environment anomalies through the joint training mechanism of interference anomaly labeling and real anomalies, and enhances the model's generalization ability for unseen interference types. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a simplified diagram of the overall algorithm flow;

[0051] Figure 2 A simplified flow chart of the module for generating heat maps of suspicious areas;

[0052] Figure 3 A simplified flow chart of the module for generating heat maps of suspicious areas. DETAILED DESCRIPTION

[0053] The following is a detailed description of the embodiments of the present invention. Although the present invention will be described and illustrated in conjunction with certain specific embodiments, it should be noted that the present invention is not limited to these embodiments. On the contrary, modifications or equivalent substitutions of the present invention are intended to fall within the scope of the claims of the present invention.

[0054] In addition, in order to better illustrate the present invention, numerous specific details are given in the following detailed description. It will be understood by those skilled in the art that the present invention can also be implemented without these specific details.

[0055] Example 1

[0056] The present invention provides a method for detecting anomalies of positive samples in an open environment based on heat maps and target detection. The solution includes the following contents:

[0057] 1. Basic training dataset construction method

[0058] 1.1 Select N images of the same point at different times, one of which is used as the test image and N-1 images as the base image.

[0059] 1.2 Use SuperPoint+SuperGlue to detect key points of the base map based on the test map in the dataset constructed in 1.1, obtain the correction transformation matrix, and correct the base map.

[0060] 1.3 Duplicate the dataset corrected in 1.2 to generate two datasets, dataset A and dataset B:

[0061] 1.3.1 For dataset A:

[0062] 1.3.1.1 Case 1: Anomalies are randomly generated only on the test image. The anomalies generated in this type are the ones that really need to be detected, and the location of the anomaly is marked with an outer bounding rectangle.

[0063] 1.3.1.2 Case 2: The same anomaly is randomly generated at the same location on the detection map and the base map. This type of anomaly does not require detection.

[0064] 1.3.1.3 Dataset A contains two data subsets: A_test map and A_base map.

[0065] 1.3.2 Keep the dataset B as it is. The dataset B contains two data subsets: B_test map and B_base map.

[0066] 2. Algorithm construction plan

[0067] 2.1 Suspicious Area Heat Map Generation Module

[0068] 2.1.1 Fine-tuning the feature extraction network

[0069] 2.1.1.1 Select the B_basemap part of dataset B as the training set.

[0070] 2.1.1.2 Use the feature extraction network (ResNet50) as the backbone of the Siamese network and perform 5-10 rounds of fine-tuning using the B_base map data from dataset B. This minimizes the difference in features extracted by the feature extraction network for images with different interference conditions at the same point. This reduces the impact of some environmental noise on detection accuracy.

[0071] 2.1.2 Using the feature extraction network fine-tuned in 2.1.1, extract semantic features from the base image and test image, obtaining base image semantic features and test image semantic features, respectively. Calculate the Mahalanobis distance between the test image semantic features and all corresponding base image semantic features, and use the minimum Mahalanobis distance value as the corresponding position value in the heat map.

[0072] 3. Anomaly Detection Network

[0073] 3.1 Constructing anomaly detection network

[0074] 3.1.1 Modify yolov5 and copy the original yolov5 backbone network to obtain A_backbone network and B_backbone network. The test image and base map share the A_backbone network, and the suspicious area heat map uses the B_backbone network.

[0075] 3.1.2 Perform the following operations on the three outgoing layers of the original yolov5 backbone network:

[0076] 3.1.2.1 The test image and the base image are passed through the A_backbone network to obtain two semantic features. These two semantic features are then subtracted to obtain the difference features. At the three extraction layers of the A_backbone network, the difference features A, B, and C between the test image and the base image are obtained.

[0077] 3.1.2.2 The suspicious area heat map is passed through the B_backbone network, and features are added at the three extraction layers and the corresponding difference features A, difference features B, and difference features C. The resulting features are used as the final output features and passed to the subsequent detection heads 1, 2, and 3.

[0078] 3.2 Constructing anomaly detection training set

[0079] 3.2.1 On dataset A, use the suspicious area heat map generation module to obtain the suspicious area heat map of each point test map and base map in dataset A, thereby obtaining heat map set C.

[0080] 3.2.2 Merge dataset A and heat map set C to obtain the anomaly detection training set D. The training set D includes three data subsets: D_test map subset, D_base map subset, and D_suspicious area heat map subset.

[0081] 3.3 Training anomaly detection network

[0082] 3.3.1 Feed the D_test map and D_base map of the same point into the A_backbone network, and feed the D_suspicious area heat map into the B_backbone network. Complete the training of the anomaly detection network in this way.

[0083] 4 Specific steps for algorithm implementation

[0084] 4.1 The test image_a and the corresponding base image_a are sent to the suspicious area heat map generation module. The suspicious area heat map_a is obtained by obtaining the minimum Mahalanobis distance between the corresponding positions of the semantic features of the test image and the semantic features of the base image.

[0085] 4.2 The test image_a and the corresponding base image_a are respectively passed through the Yolov5 A_backbone network to obtain the test image semantic feature_a and the base image semantic feature_a. The test image semantic feature_a is subtracted from the base image semantic feature_a to obtain the difference feature_a. The difference feature_a and the suspicious area heat map_a are then passed through the Yolov5 B_backbone network to obtain the suspicious area heat map feature_b, which is added to obtain the fused feature_b. The fused feature_b is then sent to the subsequent detection head of Yolov5 for anomaly detection and localization.

Claims

1. A method for detecting anomalies in positive samples in an open environment based on heatmaps and target detection, comprising the following steps: Step 1. Basic training dataset construction: 1.a. Collect N images of the same point at different times, select one as the test image, and the remaining N-1 images as the base image; 1.b. Detect and correct key points on the test image and base image to generate a corrected dataset; 1.c. Copy the corrected dataset to generate dataset A and dataset B, where: Dataset A adds anomaly areas to the test image and marks the outer bounding rectangle of the anomaly areas, or generates the same anomaly in the same area of ​​the test image and the base image; Dataset B keeps the original images without adding abnormalities; Step 2. Generate heat map of suspicious areas: 2.a. Use the Siamese network structure to fine-tune the feature extraction network to reduce the difference in features extracted from images with different environmental noise at the same point; 2.b. Extract semantic features from the test image and base image, calculate the minimum Mahalanobis distance between the semantic features of the test image and each base image, and generate a heat map; Step 3. Anomaly detection network training and detection: 3.a. Build a dual-backbone object detection network, with the first backbone network processing the difference features between the test image and the base image, and the second backbone network processing the heat map features; 3.b. Fusion of differential features and heat map features at multiple levels, input into the detection head for anomaly location and classification; 3.c. Use the heatmap generated by dataset A and the anomalies to train the anomaly detection network.

2. The method according to claim 1, wherein the key point detection and correction in step 1.b comprises: The SuperPoint algorithm is used to detect the key points of the test image and the base image. The key point pairs are matched using the SuperGlue algorithm. The correction transformation matrix is ​​calculated based on the matching results, and the base image is affine transformed to align it with the test image.

3. The method according to claim 1, characterized in that The generation of the dataset A includes at least one of the following situations: Only random abnormal areas are generated on the test image and marked as abnormalities to be detected; The same abnormal area is generated at the same position of the test image and the base image, and marked as an interference anomaly that does not need to be detected.

4. The method according to claim 1, wherein The fine-tuning feature extraction network in step 2.a includes: ResNet50 is used as the backbone of the twin network, and the base map of dataset B is used as the training set. The network parameters are adjusted by minimizing the feature differences of images under different interference conditions of the same scene.

5. The method according to claim 1, wherein The method for generating a heat map in step 2.b includes: The Mahalanobis distance between the semantic features of the test image and each base image is calculated pixel by pixel, and the minimum distance value of the corresponding position of each base image is taken as the pixel value of the heat map. The high-value area in the heat map indicates potential anomalies.

6. The method according to claim 1, characterized in that The dual-backbone target detection network in step 3.a is based on the improved YOLOv5 architecture and includes: Copy the YOLOv5 backbone network into the first backbone network and the second backbone network; The first backbone network extracts the difference features between the test image and the base image; The second backbone network extracts heat map features and performs feature addition and fusion with difference features at multiple levels.

7. The method according to claim 6, characterized in that The acquisition of the difference feature includes: The test image and the base image are input into the first backbone network respectively to obtain multi-layer semantic features, and the corresponding layer features are subtracted to generate difference features A, B, and C.

8. The method according to claim 1, characterized in that The construction of the anomaly detection training set in step 3.c includes: A heatmap set is generated for the test image and base map in dataset A, and the test image, base map, heatmap and anomaly annotations are combined into a training set for end-to-end training of the anomaly detection network.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program, and when the processor executes the program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium, characterized in that Computer instructions are stored, and when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images