Image change detection method and system
The integration of target detection and twin network segmentation in image change detection methods reduces data set collection and annotation, and computational requirements, enhancing efficiency and applicability.
Patent Information
- Application Number
- CN202411168666.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-12-17
AI Technical Summary
Existing image change detection methods require large amounts of image data sets, extensive annotation, and high computational resources due to pixel-by-pixel comparison and feature extraction, leading to inefficiencies.
Integrates target detection and twin network segmentation by sampling and preprocessing datasets, extracting features, and fusing them to reduce data collection and annotation work, using a combined loss function for training.
Reduces data set collection and annotation work while lowering computational demands, expanding application scenarios by leveraging both target detection and twin network segmentation for image change detection.
Smart Images
Figure CN119152182A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image detection, and in particular to an image change detection method and system. Background Art
[0002] Image change detection is based on two images taken at different times (i.e., a reference image and a test image), and identifies changes in the same area in the two images by comparison. Thus, by comparing with the reference image under normal circumstances, anomalies in the image can be detected and discovered, which can be used to detect moving objects.
[0003] On this basis, the image change detection method in the existing technology is mainly based on image difference and pixel-by-pixel comparison, and combines improved solutions such as feature point extraction and image registration to complete image change detection. However, this method requires the collection of a large number of image data sets, as well as annotation and other work. At the same time, when performing comparative detection, the amount of calculation is large and the energy consumption is high. Summary of the invention
[0004] The purpose of the present invention is to provide an image change detection method and system, which can integrate the advantages of target detection and twin network segmentation, greatly reduce the work of data set collection and annotation, and reduce the amount of calculation.
[0005] To achieve the above object, the present invention provides the following technical solutions:
[0006] On the one hand, a method for detecting image changes is provided, which comprises the following steps:
[0007] Sampling the twin network dataset and the target detection dataset, and preprocessing the sampled data;
[0008] Extract the twin network data features and target detection data features after data preprocessing, and perform feature fusion on the twin network data features and target detection data features respectively;
[0009] Process the twin network data fusion features and target detection data fusion features respectively, and output the corresponding processing results;
[0010] Repeat the above steps to train the model and output the image change detection model;
[0011] Image changes are detected through the image change detection model, and the image change results are output.
[0012] Preferably, data sampling of the twin network dataset and the target detection dataset comprises the following steps:
[0013] Read the twin network dataset and target detection dataset respectively;
[0014] Randomly sample data from the twin network dataset and randomly sample data from the target detection dataset;
[0015] The sampled twin network data and target detection data are packaged into at least one batch, and in each batch, the quantity ratio of the twin network data and the target detection data meets the preset conditions.
[0016] Preferably, extracting the twin network data features and target detection data features after data preprocessing includes the following steps:
[0017] The twin network data features that have been preprocessed are extracted through the backbone feature extraction network of the twin network, and the target detection data features that have been preprocessed are extracted through the backbone feature extraction network of the target detection network.
[0018] Preferably, the twin network data fusion features and the target detection data fusion features are processed separately, and the corresponding processing results are output, including the following steps:
[0019] The twin network data fusion features obtained by feature fusion are segmented to obtain the twin network output results, and target detection is performed on the target detection data fusion features obtained by feature fusion to obtain the target detection output results.
[0020] Preferably, segmenting the twin network data fusion features obtained by feature fusion to obtain the twin network output result includes the following steps:
[0021] The twin network segmentation head mHead is used to segment the twin network data fusion features and perform binary classification on the pixels to generate a binary mask image as the output result of the twin network.
[0022] Preferably, performing target detection on the target detection data fusion features obtained by feature fusion to obtain a target detection output result comprises the following steps:
[0023] The target detection network detection head dHead is used to perform target detection on the target detection data fusion features and output the target detection output results.
[0024] Preferably, the image change detection method further comprises: constructing a total loss function L total , and L total =α*L mask +β*L reg +λ*L cls , and the training is supervised by the total loss function, where L mask is the loss function of the twin network; L reg is the regression loss function; Lcls is the classification loss function; α, β, λ are adjustment coefficients.
[0025] Preferably, detecting image changes by using an image change detection model and outputting image change results comprises the following steps:
[0026] Inputting a reference image and an image to be tested into the image change detection model, and performing image preprocessing on the reference image and the image to be tested respectively;
[0027] The features of the reference image that has undergone image preprocessing and the features of the image to be tested that has undergone image preprocessing are respectively extracted through a backbone feature extraction network;
[0028] The reference image features and the image features to be tested are respectively fused through the twin network of the image change detection model to obtain a first image fusion feature;
[0029] The first image fusion feature is segmented to obtain a first image change output result.
[0030] Preferably, detecting image changes by using an image change detection model and outputting image change results also includes the following steps:
[0031] The target detection network of the image change detection model performs feature fusion on the reference image features and the image features to be tested to obtain a second image fusion feature;
[0032] Perform target detection on the second image fusion features to obtain a second image change output result.
[0033] On the other hand, there is also provided an image change detection system, comprising:
[0034] A data processing unit, which is used to sample the twin network data set and the target detection data set respectively, and to perform data preprocessing on the sampled twin network data and target detection data;
[0035] A feature extraction unit, which is used to extract the features of the twin network data after data preprocessing, and to extract the features of the target detection data after data preprocessing;
[0036] A feature fusion unit, which is used to fuse all extracted twin network data features and all extracted target detection data features respectively, so as to obtain twin network data fusion features and target detection data fusion features respectively;
[0037] A fusion feature processing unit, which is used to perform segmentation processing on the fusion features of the twin network data to obtain the twin network output result, and to perform target detection on the fusion features of the target detection data to obtain the target detection output result;
[0038] The detection unit is used to detect image changes according to the image change detection model output after the training is completed, and output the image change detection result.
[0039] In summary, the present invention has the following beneficial effects compared with the prior art:
[0040] The present invention combines the advantages of target detection and twin network segmentation, which can greatly reduce the work of data set collection and annotation. At the same time, when performing image change detection, the reference image features and the image features to be tested can be used as the basic features of target detection and twin network segmentation at the same time, further reducing the amount of calculation and expanding the application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 is a flowchart of the steps of the image change detection method of the present invention;
[0042] Figure 2 The binary mask image obtained in step S6 of the present invention;
[0043] Figure 3 The target detection output result obtained in step S6 of the present invention;
[0044] Figure 4 It is a structural schematic diagram of the image change detection system of the present invention. DETAILED DESCRIPTION
[0045] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0046] Example 1
[0047] like Figure 1 The present embodiment provides an image change detection method, which comprises the following steps:
[0048] S1. Construct a training data set, wherein the training data set includes a Siamese Network data set and a target detection data set;
[0049] Specifically, the twin network dataset includes several reference images and several test images, wherein each reference image may correspond to one or more test images, and the test image has image changes in the same area compared with the corresponding reference image, and the image changes include target disappearance, target movement, target appearance, etc., and the test images with different image changes can be selected according to the actual training requirements;
[0050] The target detection data set includes a movable or immovable target image, and the target image has a target detection frame for identifying the target, wherein the target detection frame can be generated based on a target detection algorithm such as a YOLO series algorithm (such as YOLOv5, YOLOv8, etc.), an SSD algorithm (Single Shot MultiBox Detector), or RetinaNet, and the target in the image is annotated in a VOC format. In this embodiment, the target detection data can be obtained based on various open source databases, and the annotated target object can be a common category with high detection and recognition accuracy and precision, such as a human body, a vehicle, an animal, a building, etc.;
[0051] S2. Use modules such as DataLoader to sample the twin network data set and the target detection data set respectively, and package the sampled twin network data and target detection data into at least one batch;
[0052] Specifically, step S2 includes the following steps:
[0053] Read the twin network dataset and target detection dataset respectively;
[0054] Randomly sample data from the twin network dataset through modules such as DataLoader, and randomly sample data from the target detection dataset;
[0055] The sampled twin network data and target detection data are packaged into at least one batch, and in each batch, the ratio of the twin network data and the target detection data meets the preset conditions. For example, in this embodiment, the ratio of the sampled twin network data and the target detection data is not less than 1:2 or 2:1, etc. The ratio condition can be set according to actual needs; at the same time, there can be multiple batches in this embodiment, and the specific number can be set according to factors such as the GPU video memory size;
[0056] S3. Using a data loader (DataLoader) and other modules to perform data preprocessing on the twin network data and target detection data in at least one batch (batch); in this embodiment, it is preferred to perform data preprocessing on the twin network data and target detection data in multiple batches (batch);
[0057] The preprocessing of the twin network data includes one or more operations such as resizing, random flipping, and normalizing the image (including the reference image and / or the test image);
[0058] The preprocessing of target detection data includes one or more operations such as Mosaic data enhancement, Mixup data enhancement, random color change, resize, random flip, and normalization.
[0059] Therefore, the data quality can be improved by preprocessing the sampled data in the batch, which helps to improve the accuracy and performance of the decision-making process during subsequent model training.
[0060] S4. Extracting the twin network data features in each batch that has undergone data preprocessing through the backbone feature extraction network (Backbone) of the twin network, and extracting the target detection data features in each batch that has undergone data preprocessing through the backbone feature extraction network (Backbone) of the target detection network;
[0061] In this embodiment, the backbone feature extraction network (Backbone) includes one or more network models such as Resnet, VGG, SwinTransformer, etc.; the twin network data features include reference image features and test image features, and the target detection data features include one or more of the target's texture, shape, color, size, position, etc.;
[0062] S5. All twin network data features extracted from the batches that have undergone data preprocessing and all target detection data features extracted from the batches that have undergone data preprocessing are respectively fused to obtain twin network data fusion features and target detection data fusion features respectively;
[0063] Among them, the feature fusion of all the twin network data features extracted from the batch after data preprocessing includes: using the temporal features (MTF, Merge temporal features) merging module to fuse all the benchmark image features and test image features to obtain the twin network data fusion features;
[0064] The feature fusion of all target detection data features extracted from the batch after data preprocessing includes: using a network model such as Feature Pyramid Networks (FPN) to fuse all target detection data features to obtain target detection data fusion features; preferably, the Feature Pyramid Network model (FPN) includes one or more of Pyramid Attention Network (PAN), Bi-directional Feature Pyramid Networks (BiFPN), etc.;
[0065] S6. Segment the twin network data fusion features to obtain the twin network output results, and perform target detection on the target detection data fusion features to obtain the target detection output results;
[0066] Among them, the twin network data fusion features are segmented to obtain the twin network output results, including:
[0067] The twin network segmentation head mHead is used to segment the twin network data fusion features; wherein the twin network segmentation head mHead can be one or more of DeepLabHead and FCNHead;
[0068] The segmented twin network data fusion features are upsampled to the input image size, and the pixels are classified into binary values, so that the pixel values of the changed part of the image are recorded as 1, and the pixel values of the unchanged part of the image are recorded as 0, so as to generate a binary mask image (such as Figure 2 shown);
[0069] The binary mask image is output as the output result of the twin network;
[0070] Perform target detection on the target detection data fusion features to obtain target detection output results, including:
[0071] The target detection network detection head dHead is used to perform target detection on the target detection data fusion features;
[0072] Output the target detection output result, and the target detection output result includes one or more of the target detection frame, target category, and score (such as Figure 3 shown);
[0073] S7, repeat steps S2-S6 to train the model and construct the total loss function L at the same time total , and L total =α*L mask +β*L reg +λ*L cls , and supervise the training through the total loss function until the training meets the preset conditions, and then output the image change detection model;
[0074] Among them, L mask is the loss function of the twin network, which can be cross entropy (CE), and only needs to supervise the binary classification result of the output mask image; the loss function of the target detection network includes the regression loss function and the classification loss function, L reg That is, the regression loss function. In this embodiment, GIoU can be used as the regression loss function L reg , L cls That is, the classification loss function. In this embodiment, Focal Loss can be used as the classification loss function L cls ; α, β, λ are all adjustment coefficients and can be set according to actual needs.
[0075] Therefore, the above total loss function L total As a supervisory signal for network model training to improve the model training effect; at the same time, after the model training is completed, the model parameters are solidified so that the final image change detection model still includes the twin network and the target detection network;
[0076] S8, detecting image changes through an image change detection model and outputting image change results, which specifically includes the following steps:
[0077] Inputting a reference image and an image to be tested into the image change detection model, and performing image preprocessing on the reference image and the image to be tested respectively, wherein the image preprocessing includes one or more operations such as resizing and normalizing the image;
[0078] The backbone feature extraction network (Backbone) of the image change detection model is used to extract the features of the reference image after image preprocessing and the features of the image to be tested after image preprocessing respectively;
[0079] The reference image features and the image features to be tested are respectively fused by the twin network of the image change detection model and the target detection network to obtain the first image fusion feature and the second image fusion feature respectively; specifically, in this embodiment, the twin network of the image change detection model uses the merged temporal feature (MTF) module to fuse the reference image features and the image features to be tested to obtain the first image fusion feature; the target detection network of the image change detection model uses the feature pyramid (FPN) module to fuse the reference image features and the image features to be tested to obtain the second image fusion feature;
[0080] The first image fusion feature is segmented to obtain a first image change output result, and the second image fusion feature is subjected to target detection to obtain a second image change output result; specifically, in this embodiment, a twin network segmentation head mHead is used to segment the first image fusion feature to generate a binary mask image, which is the first image change output result, and a target detection network detection head dHead is used to perform target detection on the second image fusion feature to obtain a second image change output result, and the second image change output result includes one or more of a target detection box, a target category, and a score; wherein, the area where the image to be tested has changed relative to the reference image can be directly known from the first image output result (i.e., the binary mask image), and whether the target appears / disappears in the image to be tested relative to the reference image, and whether the target position has changed, etc. can be known from the second image change output result;
[0081] The first image change output result and the second image change output result are combined to obtain a final image change detection result, and output the result;
[0082] Therefore, in this embodiment, small targets and noise can be filtered through the binary mask image as the first image change output result to display the changed image area, and the change of the target can be obtained through the second image change output result. After the two are fused, the accuracy of the final image change detection result can be guaranteed.
[0083] When using the target detection network alone for image change detection, its application scenarios are limited because target detection can only label common categories and cannot label all target types. When using the twin network alone to obtain image change results, there are defects such as the difficulty in obtaining data sets (because paired reference images and images to be tested are required), difficulty in labeling, and complex calculations caused by segmentation.
[0084] This embodiment combines the advantages of target detection and twin network segmentation. It uses the target detection network for common target categories and uses the twin network segmentation for uncommon target categories and scenarios to obtain image change results. This can expand the application scenarios and greatly reduce the data set collection and annotation work. At the same time, when performing image change detection through the image change detection model, the baseline image features and the image features to be tested can be used as the basic features of target detection and twin network segmentation at the same time, further reducing the amount of calculation, while having the advantages of target detection and twin network segmentation.
[0085] Embodiment 2:
[0086] This embodiment provides an image change detection system, which is used to implement the image change detection method described in Embodiment 1. Figure 4 As shown, the image change detection system includes:
[0087] A training data set construction unit 1, which is used to construct a training data set including a Siamese Network data set and a target detection data set;
[0088] A data processing unit 2 is used to perform data sampling on the twin network data set and the target detection data set respectively, and perform data preprocessing on the sampled twin network data and target detection data, wherein the data sampling method is the same as step S2, and the data preprocessing method is the same as step S3;
[0089] A feature extraction unit 3 is used to extract the twin network data features in each batch that has undergone data preprocessing through the backbone feature extraction network (Backbone) of the twin network, and to extract the target detection data features in each batch that has undergone data preprocessing through the backbone feature extraction network (Backbone) of the target detection network. The data feature extraction method is the same as step S4;
[0090] A feature fusion unit 4 is used to fuse all twin network data features extracted from batches that have undergone data preprocessing and all target detection data features extracted from batches that have undergone data preprocessing, respectively, to obtain twin network data fusion features and target detection data fusion features, and the feature fusion method is the same as step S5;
[0091] A fusion feature processing unit 5 is used to perform segmentation processing on the fusion features of the twin network data to obtain the twin network output result, and to perform target detection on the fusion features of the target detection data to obtain the target detection output result. The fusion feature processing is the same as step S6;
[0092] Loss function construction unit 6, which is used to construct the total loss function L total , so as to supervise the training through the total loss function;
[0093] The detection unit 7 is used to detect image changes according to the image change detection model output after the training is completed, and output the image change detection result, the content of which is the same as step S8.
[0094] To sum up, the present invention integrates the advantages of target detection and twin network segmentation. It uses target detection network for common target categories and uses twin network segmentation for uncommon target categories and scenes to obtain image change results, thereby expanding the application scenarios and greatly reducing the work of data set collection and annotation. At the same time, when performing image change detection, the baseline image features and the image features to be tested can be used as the basic features of target detection and twin network segmentation at the same time, further reducing the amount of calculation.
[0095] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for detecting image changes, characterized in that: The steps include: Sampling the twin network dataset and the target detection dataset, and preprocessing the sampled data; Extract the twin network data features and target detection data features after data preprocessing, and perform feature fusion on the twin network data features and target detection data features respectively; Process the twin network data fusion features and target detection data fusion features respectively, and output the corresponding processing results; Repeat the above steps to train the model and output the image change detection model; Image changes are detected through the image change detection model, and the image change results are output.
2. The image change detection method according to claim 1, characterized in that: Data sampling for the twin network dataset and the target detection dataset includes the following steps: Read the twin network dataset and target detection dataset respectively; Randomly sample data from the twin network dataset and randomly sample data from the target detection dataset; The sampled twin network data and target detection data are packaged into at least one batch, and in each batch, the quantity ratio of the twin network data and the target detection data meets the preset conditions.
3. The image change detection method according to claim 1, characterized in that: Extracting the twin network data features and target detection data features after data preprocessing includes the following steps: The twin network data features that have been preprocessed are extracted through the backbone feature extraction network of the twin network, and the target detection data features that have been preprocessed are extracted through the backbone feature extraction network of the target detection network.
4. The image change detection method according to claim 1, characterized in that: The twin network data fusion features and target detection data fusion features are processed separately, and the corresponding processing results are output, including the following steps: The twin network data fusion features obtained by feature fusion are segmented to obtain the twin network output results, and target detection is performed on the target detection data fusion features obtained by feature fusion to obtain the target detection output results.
5. The image change detection method according to claim 4, characterized in that: The twin network data fusion features obtained by feature fusion are segmented to obtain the twin network output results, including the following steps: The twin network segmentation head mHead is used to segment the twin network data fusion features and perform binary classification on the pixels to generate a binary mask image as the output result of the twin network.
6. The image change detection method according to claim 4, characterized in that: Performing target detection on the target detection data fusion features obtained by feature fusion to obtain the target detection output result includes the following steps: The target detection network detection head dHead is used to perform target detection on the target detection data fusion features and output the target detection output results.
7. The image change detection method according to claim 1, characterized in that: The image change detection method further includes: constructing a total loss function L total , and L total =α*L mask +β*L reg +λ*L cls , and the training is supervised by the total loss function, where L mask is the loss function of the twin network; L reg is the regression loss function; L cls is the classification loss function; α, β, λ are adjustment coefficients.
8. The image change detection method according to claim 1, characterized in that: The image change detection model is used to detect image changes and output image change results, including the following steps: Inputting a reference image and an image to be tested into the image change detection model, and performing image preprocessing on the reference image and the image to be tested respectively; The features of the reference image that has undergone image preprocessing and the features of the image to be tested that has undergone image preprocessing are respectively extracted through a backbone feature extraction network; The reference image features and the image features to be tested are respectively fused through the twin network of the image change detection model to obtain a first image fusion feature; The first image fusion feature is segmented to obtain a first image change output result.
9. The image change detection method according to claim 8, characterized in that: The image change detection model is used to detect image changes and output image change results, which also includes the following steps: The target detection network of the image change detection model performs feature fusion on the reference image features and the image features to be tested to obtain a second image fusion feature; Perform target detection on the second image fusion features to obtain a second image change output result.
10. An image change detection system, characterized in that: include: A data processing unit, which is used to sample the twin network data set and the target detection data set respectively, and to perform data preprocessing on the sampled twin network data and target detection data; A feature extraction unit, which is used to extract the features of the twin network data after data preprocessing, and to extract the features of the target detection data after data preprocessing; A feature fusion unit, which is used to fuse all extracted twin network data features and all extracted target detection data features respectively, so as to obtain twin network data fusion features and target detection data fusion features respectively; A fusion feature processing unit, which is used to perform segmentation processing on the fusion features of the twin network data to obtain the twin network output result, and to perform target detection on the fusion features of the target detection data to obtain the target detection output result; The detection unit is used to detect image changes according to the image change detection model output after the training is completed, and output the image change detection result.