Automated labeling method and system for change detection images

Abstract

The application discloses a kind of automatic change detection image labeling method and system, it is related to image processing technical field.The method comprises the following steps: the feature extraction of pixel change and moving object change in double-phase image set is carried out, and a plurality of change labeling maps are obtained;The feature extraction of focus object in double-phase image set is carried out, and a plurality of object labeling maps are obtained;Each object labeling map is operated with each change labeling map, and object change labeling map set is generated;The target noise area of each object change labeling map in object change labeling map set is found, the target noise area is repaired, and the repaired object change labeling map set is obtained;New unlabeled data set is acquired, and the semi-supervised labeling model is trained according to new unlabeled data set and repaired object change labeling map set, and the trained semi-supervised labeling model is obtained;According to the trained semi-supervised labeling model, double-phase traffic scene image is automatically labeled, and semi-supervised labeling data set is generated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a change detection image automatic labeling method and system. BACKGROUND

[0002] Change detection technology is an operation of quantitatively analyzing the surface changes of phenomena or objects in different periods. At present, change detection technology is mainly applied in remote sensing image scenes, such as urban expansion, deep forest felling and post-disaster assessment. The development stage of change detection technology research can be divided into three stages: pixel-level change detection, object-level change detection and deep learning-based change detection. Due to the influence of multiple factors such as the refraction angle of light, climate and sensor, change detection and its labeling become more difficult, and strict and complex preprocessing is required.

[0003] Most of the traditional change detection datasets are remote sensing images, and the labeling process is to collect a large number of scene images and finely label the change areas, and the double-time scene images and the labeled images constitute the change detection dataset. The traditional labeling method has high labor cost and long cycle, resulting in low labeling efficiency, which cannot be widely applied to various scenes. SUMMARY

[0004] The purpose of the present application is to solve the technical problem of low labeling efficiency of the traditional method. In order to achieve the above purpose, the present application provides a change detection image automatic labeling method and system.

[0005] The present application provides a change detection image automatic labeling method, comprising:

[0006] Obtaining an unsupervised image set, and dividing the unsupervised image set into double-time image sets to form a double-time image set;

[0007] Extracting features of pixel changes and moving object changes in the double-time image set to obtain a plurality of change labeling maps;

[0008] Extracting features of the objects of interest in the double-time image set to obtain a plurality of object labeling maps;

[0009] Performing AND operation on each of the object labeling maps and each of the change labeling maps to generate an object change labeling map set;

[0010] Finding a target noise area of each object change labeling map in the object change labeling map set, repairing the target noise area to obtain a repaired object change labeling map set;

[0011] acquire a new unlabeled dataset, train a semi-supervised labeling model according to the new unlabeled dataset and the repaired object change labeling atlas, and obtain a trained semi-supervised labeling model;

[0012] automatically label the dual-time-phase traffic scene images according to the trained semi-supervised labeling model, and generate a semi-supervised labeling dataset.

[0013] In one embodiment, after the dual-time-phase traffic scene images are automatically labeled according to the trained semi-supervised labeling model to generate a semi-supervised labeling dataset, the method further comprises:

[0014] train a supervised labeling model according to the repaired object change labeling atlas, and obtain an initial supervised labeling model;

[0015] input the semi-supervised labeling dataset into the initial supervised labeling model, and output a change detection result atlas;

[0016] correctively compare the change detection result atlas with the repaired object change labeling atlas, iteratively optimize the initial supervised labeling model, and obtain a trained supervised labeling model;

[0017] automatically label each traffic scene image according to the trained supervised labeling model, and generate a supervised labeling dataset.

[0018] In one embodiment, the feature extraction of the pixel change and the moving object change in the dual-time-phase image set to obtain a plurality of change labeling maps comprises:

[0019] extract and process the change regions of the dual-time-phase image set according to the frame difference method, and obtain a pixel-level change map;

[0020] extract and process the change regions of the dual-time-phase image set according to the background difference method, and obtain a moving object-level change map;

[0021] perform an AND operation on the pixel-level change map and the moving object-level change map, and obtain the change labeling map.

[0022] In one embodiment, the feature extraction of the focus object in the dual-time-phase image set to obtain a plurality of object labeling maps comprises:

[0023] segment each dual-time-phase image in the dual-time-phase image set according to a semantic segmentation model, and obtain a corresponding set of focus object mask maps;

[0024] convert each set of focus object mask maps into a binary map, and obtain a corresponding set of focus object binary maps;

[0025] Map each of the object binary image to the same image by pixel-by-pixel OR operation to generate the object label image.

[0026] In one embodiment, the target noise region of each object change label image in the object change label image set is found, and the target noise region is repaired to obtain a repaired object change label image set, comprising:

[0027] An independent noise region area threshold and an independent hole region area threshold are set.

[0028] If the area of the independent noise region in the object change label image is less than the independent noise region area threshold, the independent noise region is the target noise region, and the independent noise region is removed to obtain a repaired object change label image.

[0029] Alternatively, if the area of the independent hole region in the object change label image is less than the independent hole region area threshold, the independent hole region is the target noise region, and the independent hole region is filled with holes to obtain the repaired object change label image.

[0030] The present application provides an automatic annotation system for change detection images, comprising:

[0031] An image acquisition module is configured to acquire an unsupervised image set and divide the unsupervised image set into a dual-phase image set.

[0032] A change label image extraction module is configured to extract features of pixel changes and motion object changes in the dual-phase image set to obtain a plurality of change label images.

[0033] An object label image extraction module is configured to extract features of objects of interest in the dual-phase image set to obtain a plurality of object label images.

[0034] An object change image acquisition module is configured to perform AND operation between each of the object label images and each of the change label images to generate an object change label image set.

[0035] A repair module is configured to find a target noise region of each object change label image in the object change label image set and repair the target noise region to obtain a repaired object change label image set.

[0036] A semi-supervised annotation model training module is configured to acquire a new unlabeled data set, train a semi-supervised annotation model according to the new unlabeled data set and the repaired object change label image set, and obtain a trained semi-supervised annotation model.

[0037] The semi-supervised labeling module is configured to automatically label the dual-time-phrase traffic scene images according to the trained semi-supervised labeling model to generate a semi-supervised labeling dataset.

[0038] In one embodiment, the system further comprises:

[0039] The initial model training module is configured to train a supervised labeling model according to the repair object change labeling image set to obtain an initial supervised labeling model.

[0040] The model output module is configured to input the semi-supervised labeling dataset into the initial supervised labeling model to output a change detection result image set.

[0041] The supervised labeling model training module is configured to compare and correct the change detection result image set with the repair object change labeling image set to iteratively optimize the initial supervised labeling model to obtain a trained supervised labeling model.

[0042] The supervised labeling module is configured to automatically label each traffic scene image according to the trained supervised labeling model to generate a supervised labeling dataset.

[0043] In one embodiment, the change labeling image extraction module comprises:

[0044] The first change image extraction module is configured to perform change region extraction processing on the dual-time-phrase image set according to a frame difference method to obtain a pixel-level change image.

[0045] The second change image extraction module is configured to perform change region extraction processing on the dual-time-phrase image set according to a background difference method to obtain a moving object-level change image.

[0046] The AND operation module is configured to perform AND operation on the pixel-level change image and the moving object-level change image to obtain the change labeling image.

[0047] In one embodiment, the object labeling image extraction module comprises:

[0048] The semantic segmentation module is configured to segment each dual-time-phrase image in the dual-time-phrase image set according to a semantic segmentation model to obtain a corresponding set of attention object mask images.

[0049] The binary image conversion module is configured to convert each set of the attention object mask images into a binary image to obtain a corresponding set of attention object binary images.

[0050] The mapping module is configured to perform pixel-by-pixel OR operation on each set of the attention object binary images to map them into the same image to generate the object labeling image.

[0051] In one embodiment, the repairing module comprises:

[0052] a threshold setting module for setting an independent noise region area threshold and an independent hole region area threshold;

[0053] a noise repairing module for, if the area of an independent noise region in the object change annotation map is less than the independent noise region area threshold, regarding the independent noise region as the target noise region and removing the independent noise region to obtain a repaired object change annotation map; or

[0054] a hole repairing module for, if the area of an independent hole region in the object change annotation map is less than the independent hole region area threshold, regarding the independent hole region as the target noise region and performing hole filling on the independent hole region to obtain the repaired object change annotation map.

[0055] In the above-mentioned automatic annotation method and system for change detection images, the unsupervised image set is divided to form a double-time-phase image set. For the double-time-phase images, first, the change regions are extracted from the two angles of pixel points and moving objects, which can reduce the error influence existing in different angles, suppress the interference of static background, and improve the performance of dynamic change detection. Then, the change annotation map and the object annotation map are fused and matched to generate an object change annotation map in a traffic scene. Further, the object change annotation map in the traffic scene is perfected and the noise problems existing in the annotation are repaired, so as to improve the annotation accuracy and form a repaired object change annotation map set. The repaired object change annotation map set is used as a preliminary data set, a new unlabeled data set is combined to train a semi-supervised annotation model, a trained semi-supervised annotation model is obtained, and a simple and easy-to-operate high-performance automatic traffic scene annotation tool is generated.

[0056] Further, the automatic annotation is realized through the trained semi-supervised annotation model, a large amount of automatic annotation data can be output and obtained, and a semi-supervised annotation data set is formed. Through the automatic annotation method provided by the present application, the data is annotated and verified by using a fully automatic method, the labor cost is reduced, the annotation period is shortened, and the annotation efficiency is improved. The method can be used in different change detection scenes to meet the change detection data annotation needs of various scenes. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 is a step flowchart of the automatic annotation method for change detection images provided by the present application.

[0058] Figure 2 is a structure diagram of the automatic annotation system for change detection images provided by the present application. DETAILED DESCRIPTION

[0059] The technical solutions of the present application are described in further detail below with reference to the accompanying drawings and examples.

[0060] Please refer to Figure 1 The present application provides an automatic change detection image labeling method, comprising:

[0061] S10, obtaining an unsupervised image set and performing double-phase image division on the unsupervised image set to form a double-phase image set;

[0062] S20, performing feature extraction on pixel changes and moving object changes in the double-phase image set to obtain a plurality of change labeling maps;

[0063] S30, performing feature extraction on the focus objects in the double-phase image set to obtain a plurality of object labeling maps;

[0064] S40, performing AND operation on each object labeling map and each change labeling map to generate an object change labeling map set;

[0065] S50, finding a target noise region of each object change labeling map in the object change labeling map set, repairing the target noise region, and obtaining a repaired object change labeling map set;

[0066] S60, obtaining a new unlabeled data set, training a semi-supervised labeling model according to the new unlabeled data set and the repaired object change labeling map set, and obtaining a trained semi-supervised labeling model;

[0067] S70, automatically labeling the double-phase traffic scene image according to the trained semi-supervised labeling model to generate a semi-supervised labeling data set.

[0068] In this embodiment, the unsupervised image set can be understood as a collection of original images without any labeling information, which is obtained from video information collected by a collection device. The double-phase image can be understood as an image obtained at adjacent time in the same place. In this method, the change region is extracted by a simple, fast and unsupervised method. The unsupervised method includes but is not limited to frame difference method, background difference method, etc., which extracts the change region from different angles and obtains the change labeling map and the object labeling map. Thus, the change labeling map and the object labeling map are fused to realize mutual verification and absorb the benefits brought by different methods, thereby reducing the interference of static background and improving the performance of dynamic detection.

[0069] The objects of interest in the traffic scene are vehicles and parking spaces and objects around them, and different objects of interest can also be customized according to the actual scene. The changes of the objects include moving vehicles and moving objects in the parking spaces. The change region in the traffic scene includes the change objects of interest and the change objects not of interest, and is further analyzed for the object region. The change annotation map represents the change region in the traffic scene, and the object annotation map represents the object of interest in the traffic scene. The object change annotation map in the traffic scene is generated by matching and mapping the object annotation map and the change annotation map. The change region of the object of interest is retained in the object change annotation map, and the pseudo change region of the object not of interest is discarded.

[0070] The object change annotation map of the object of interest in the dual-phase traffic scene is generated by matching and mapping each object annotation map and each change annotation map, and the operation formula is as follows:

[0071]

[0072] Where & represents an AND operation symbol, T represents a change annotation map, S represents an object annotation map, and (x, y) represents the position of each pixel point. The AND operation is performed between the pixel values at the same position in the change annotation map and the object annotation map. If the pixel values at the same position in the change annotation map and the object annotation map are both 1, the value mapped to the object change annotation map is 1. If the pixel values at the same position in the change annotation map and the object annotation map are not both 1, the value mapped to the object change annotation map is 0.

[0073] The object change annotation map of the object of interest in the traffic scene is generated by matching the object annotation map and the change annotation map, which enhances the weight of the change region of the object of interest in the traffic scene and suppresses the noise influence of the non-change region or the object region not of interest.

[0074] Since noise problems such as noise points or holes may occur in the process of generating the change annotation map and the object annotation map, the target noise region in the object change annotation map is found, the target noise region is perfected and repaired, and the further repair and optimization of the object change annotation map are realized, so that a more accurate repaired object change annotation map is obtained.

[0075] The object change annotation map set is obtained by separately extracting and fusing the change annotation map and the object annotation map of the unsupervised image set. Further, the repaired object change annotation map set is formed by repairing and optimizing the object change annotation map set, and the preliminary annotation data set is constituted.

[0076] On the basis of the repaired object change annotation map set, the semi-supervised annotation model is used to expand the data scale and increase the annotation data set, so that the demand for a large amount of annotation data set in the actual demand can be solved.

[0077] According to the semi-supervised change detection method, the training is carried out by combining a small amount of labeled data (i.e. the repaired object change annotation atlas) and a large amount of new unlabeled data set, and a trained semi-supervised annotation model is obtained. The prediction result output by the trained semi-supervised annotation model is used as a kind of annotation of the image data. The dual-phase traffic scene image is used as the input of the trained semi-supervised annotation model, and the model outputs a large amount of automatic new traffic scene annotation data set, i.e. the semi-supervised annotation data set, and the automatic annotation is completed.

[0078] The automatic annotation method of the change detection image provided by the application divides the unsupervised image set to form a dual-phase image set. For the dual-phase image, the change area is extracted from the pixel point and the moving object respectively, which can reduce the error influence existing in different angles, suppress the interference of the static background, and improve the performance of dynamic change detection. Then, the change annotation map and the object annotation map are fused and matched to generate the object change annotation map in the traffic scene. Further, the object change annotation map in the traffic scene is perfected and the noise problem existing in the annotation is repaired, and then the annotation accuracy can be improved to form the repaired object change annotation atlas. The repaired object change annotation atlas is used as the initial data set, and the semi-supervised annotation model is trained in combination with the new unlabeled data set to obtain the trained semi-supervised annotation model, and a simple and easy-to-operate high-performance automatic traffic scene annotation tool is generated. Further, the automatic annotation is realized by the trained semi-supervised annotation model, and a large amount of automatic annotation data can be output to form the semi-supervised annotation data set. By using the automatic annotation method provided by the application, the data is annotated and verified by using the fully automatic method, the labor cost is reduced, the annotation period is shortened, and the annotation efficiency is improved. The method can be used in different change detection scenes to meet the change detection data annotation demand of various scenes.

[0079] In one embodiment, S70, after the dual-phase traffic scene image is automatically annotated according to the trained semi-supervised annotation model to generate the semi-supervised annotation data set, the method further comprises:

[0080] S80, the supervised annotation model is trained according to the repaired object change annotation atlas to obtain an initial supervised annotation model;

[0081] S90, the semi-supervised annotation data set is input into the initial supervised annotation model, and a change detection result atlas is output;

[0082] S100, the initial supervised annotation model is iteratively optimized according to the comparison and correction of the change detection result atlas and the repaired object change annotation atlas to obtain a trained supervised annotation model;

[0083] S110, automatically label each traffic scene image according to the supervised labeling model trained to generate a supervised labeling data set.

[0084] In this embodiment, after the labeling tool of the semi-supervised method expands the size of the data set, the supervised algorithm is used to further improve the accuracy and performance of the model.

[0085] The preliminary labeling data set is formed by the change labeling atlas of the repair object, the model of the supervised learning method is trained, and the initial supervised labeling model is obtained. The newly formed semi-supervised labeling data set is input into the initial supervised labeling model, and the model is used to generate an accurate change detection result atlas. The change detection result atlas and the change labeling atlas of the repair object are compared and corrected, the algorithm performance is continuously iterated and optimized, and the labeling quality of the data is improved, a trained supervised labeling model is obtained, and a simple and easy-to-operate high-performance automatic labeling tool is generated. Each traffic scene image is input into the trained supervised labeling model, and an accurate change area image can be automatically generated to complete automatic labeling and obtain a supervised labeling data set.

[0086] Through the method provided by the present application, the semi-supervised learning (including but not limited to SemiCDNet, BLDNet and MTCNet, etc.) and supervised learning (including but not limited to RCDT, TINYCD and RDP-Net, etc.) methods are used to gradually generate traffic scene labeling tools, i.e. semi-supervised labeling model and supervised labeling model, from the perspective of the size of the traffic scene data set. First, the semi-supervised labeling model is used to improve the size of the traffic scene data set, and then the supervised labeling model is used to improve the accuracy of the traffic scene change detection result, and a simple and easy-to-operate high-performance automatic traffic scene labeling tool is generated. Through the automatic labeling method provided by the present application, the data of the traffic scene is labeled and verified, and the semi-supervised learning method is gradually iterated to generate a simple and easy-to-operate traffic scene change detection labeling data tool. Not only does it reduce labor costs, but it can also be used in different change detection scenarios.

[0087] In one embodiment, S20, the pixel change and moving object change in the dual-time image set are feature extracted to obtain a plurality of change labeling maps, including:

[0088] S210, the change area of the dual-time image set is extracted according to the frame difference method to obtain a pixel-level change map;

[0089] S220, the change area of the dual-time image set is extracted according to the background difference method to obtain a moving object-level change map;

[0090] S230, the pixel-level change map and the moving object-level change map are ANDed to obtain a change labeling map.

[0091] In this embodiment, the dual-time-series image set includes a plurality of dual-time-series images, each of which can be understood as being composed of two images of adjacent time points. The dual-time-series images use a frame difference method to determine moving objects in the images by calculating the difference between pixel values of the images and then determining the changes in the pixel values. When there is no change in the dual-time-series images, the corresponding pixel values are the same. When there is a change in the dual-time-series images, the corresponding pixel values change. By comparing the two images in the dual-time-series images after grayscale conversion (I k and I (k-1) ), the pixel values at each position and the pixel values that change between images show a change area on the image, which is represented as follows:

[0092] D k (x, y) | I k (x, y) - I k-1 (x, y) |.

[0093] D k (x, y) is the difference image of the dual-time-series images, and (x, y) is the pixel position on each image. The difference image D k is divided by the value range T to obtain the foreground image T k :

[0094]

[0095] where T k (x, y) is the changed binary image, and the difference result of the pixel points is greater than the value range T, which is marked as white, i.e., the foreground, and the difference result of the pixel points is less than the value range T, which is marked as black, i.e., the background. The change area required by the task can be simply and accurately extracted from the pixel-level image by the frame difference method, and the pixel-level change image T k (x, y) is obtained.

[0096] The background difference method is used to extract the change area from the dual-time-series images, and the change area is extracted from the moving objects in the images. The background difference method can be understood as follows: the background model is constructed from the previous frames of images (including but not limited to the median method and the mean modeling method, as well as the single Gaussian distribution model and the mixed Gaussian distribution model, etc.); the absolute value of the difference between the current image and the background image is obtained; the foreground and background images are constructed according to the relationship between the current absolute value image pixel points and the threshold value; and the background image is updated according to the strategy.

[0097] Specifically, k-1 images are taken from the dual-time-series image set to construct a background model to generate a background image. The background image and the current frame image are matched to obtain a deviation value D k, and further determine the moving or changing region in the image. With the passage of time, the influencing factors in the actual scene gradually increase, such as light, climate, and equipment noise, and various interferences. By constructing a background model and a background updating strategy, the influence of environmental factors in the actual scene can be solved.

[0098] In the background difference method, the current image I k (x, y) and the background image B k-1 (x, y) are subjected to difference calculation:

[0099] D k (x, y) = |I k (x, y) - B k-1 (x, y) |.

[0100] The kth image is compared with the background image updated or generated by the k-1th image pixel by pixel, and a difference image D k is constructed. The background difference method and the frame difference method both generate a binary image under the constraint of a threshold value. The changing region is extracted from the moving object of the image, and the background difference method effectively distinguishes the background and the moving object, that is, the moving object level changing map can be obtained.

[0101] The changing region is extracted from the pixel level and the moving object level of the image by the frame difference method and the background difference method, respectively. In the generated two kinds of changing region results, further analysis is performed, and the pixel level changing map and the moving object level changing map are subjected to AND operation to generate a new binary image, that is, a changing label map, which can reduce the interference such as light change between double-phase images and the difficulty of background updating. The formula for AND operation between the pixel level changing map and the moving object level changing map is as follows:

[0102]

[0103] T fd represents the pixel level changing map obtained by the frame difference method, T bs represents the moving object level changing map obtained by the background difference method, and & represents an AND operation symbol.

[0104] In the steps of the embodiment, the pixel level changing map is generated by comparing the double-phase images pixel by pixel, which can simply and accurately extract the required changing region from the pixel level image. In the process of establishing a background model and obtaining a moving foreground object, the background information is updated, which can effectively reduce the interference phenomenon such as light, and focus on the changing object. AND operation is performed between the pixel level changing map and the moving object level changing map to realize fusion, and the changing region image obtained by fusion further improves the precision of the changing region, reduces noise, effectively reduces the interference of the static background, and improves the performance of dynamic detection.

[0105] In one embodiment, S30, feature extraction is performed on the objects of interest in the dual-time-phase image set to obtain a plurality of object annotation maps, including:

[0106] S310, each dual-time-phase image in the dual-time-phase image set is segmented according to the semantic segmentation model to obtain a corresponding set of object mask maps of interest;

[0107] S320, each set of object mask maps of interest is converted into a binary map to obtain a corresponding set of object binary maps of interest;

[0108] S330, each set of object binary maps of interest is mapped into the same image through pixel-by-pixel OR operation to generate an object annotation map.

[0109] In this embodiment, the public segmentation dataset contains multiple object categories, including object categories of interest in the traffic scene. The dataset of the traffic scene constructed from the public segmentation dataset needs to be preprocessed, that is, the category labels that are not in or not needed in the traffic scene are deleted, so that the categories in the obtained dataset are all the category labels of interest, thereby constructing a new segmentation dataset. The label information in the new segmentation dataset contains the objects of the traffic scene, which are trained in the language segmentation model to generate a segmentation model, that is, a semantic segmentation model. Thus, the dual-time-phase images are segmented according to the semantic segmentation model, and the mask images are generated under the processing of the segmentation model to obtain a corresponding set of object mask maps of interest.

[0110] Each set of object mask maps of interest is converted into a binary map, in which the object region of interest is the foreground and the other regions are the background. The two object binary maps of interest containing the objects of interest in the traffic scene are operated pixel by pixel, and the formula is as follows:

[0111]

[0112] wherein B_Img A (x, y), B_Img B (x, y) represent two object binary maps of interest in each set, || represents an OR operator, that is, the pixel values at the same position in B_Img A and B_Img B are OR operated, and (x, y) represents each pixel position. By mapping each set of object binary maps of interest into the same image through pixel-by-pixel OR operation, all the object regions of interest are mapped into an image to generate an object annotation map of the dual-time-phase traffic scene. If the pixel values at the same position in the two object binary maps of interest are non-zero (i.e., the pixel value is 1), the result mapped into the object annotation map is 1. If the pixel values at the same position in the two object binary maps of interest are both 0, the result mapped into the object annotation map is 0, thereby generating an object annotation map.

[0113] In one embodiment, S50, find the target noise region of each object change annotation image in the object change annotation image set, repair the target noise region, and obtain a repaired object change annotation image set, including:

[0114] S510, set an independent noise point region area threshold and an independent hole region area threshold;

[0115] S520, if the area of the independent noise point region in the object change annotation image is less than the independent noise point region area threshold, the independent noise point region is the target noise region, and the independent noise point region is removed to obtain a repaired object change annotation image;

[0116] Or, if the area of the independent hole region in the object change annotation image is less than the independent hole region area threshold, the independent hole region is the target noise region, and the independent hole region is filled with holes to obtain a repaired object change annotation image.

[0117] In this embodiment, noise regions in a traffic scene are found in the object change annotation image. The independent noise point region area threshold and the independent hole region area threshold can be set according to actual conditions. The target noise region includes the independent noise point region and the independent hole region. When the area of the independent noise point region or the independent hole region is less than the threshold, the independent noise point region or the independent hole region is the target noise region. The independent noise point region area threshold can be 0.1% of the area of the object change annotation image. The independent hole region area threshold can be 0.1% of the area of the object change annotation image.

[0118] When repairing the target noise region in the traffic scene, the independent noise point region is removed by the removal method, and the independent hole region is completed by the hole filling method. After removing the independent noise point region and completing the independent hole region, a new binary image, i.e., a repaired object change annotation image, is generated.

[0119] Referring to Figure 2 The present application provides an automatic annotation system 100 for change detection images. The automatic annotation system 100 for change detection images includes an image acquisition module 10, a change annotation image extraction module 20, an object annotation image extraction module 30, an object change image acquisition module 40, a repair module 50, a semi-supervised annotation model training module 60, and a semi-supervised annotation module 70. The image acquisition module 10 is used to acquire an unsupervised image set and divide the unsupervised image set into a double-phase image set. The change annotation image extraction module 20 is used to extract features of pixel changes and motion object changes in the double-phase image set to obtain a plurality of change annotation images. The object annotation image extraction module 30 is used to extract features of focus objects in the double-phase image set to obtain a plurality of object annotation images.

[0120] The object change map obtaining module 40 is configured to perform an AND operation between each object annotation map and each change annotation map to generate a set of object change annotation maps. The repairing module 50 is configured to search for a target noise region of each object change annotation map in the set of object change annotation maps, repair the target noise region, and obtain a set of repaired object change annotation maps. The semi-supervised annotation model training module 60 is configured to obtain a new unlabeled data set, train a semi-supervised annotation model according to the new unlabeled data set and the set of repaired object change annotation maps, and obtain a trained semi-supervised annotation model. The semi-supervised annotation module 70 is configured to automatically annotate the dual-phase traffic scene images according to the trained semi-supervised annotation model to generate a semi-supervised annotation data set.

[0121] In this embodiment, the related description of the image obtaining module 10 can refer to the related description of S10 in the above embodiment. The related description of the change annotation map extracting module 20 can refer to the related description of S20 in the above embodiment. The related description of the object annotation map extracting module 30 can refer to the related description of S30 in the above embodiment. The related description of the object change map obtaining module 40 can refer to the related description of S40 in the above embodiment. The related description of the repairing module 50 can refer to the related description of S50 in the above embodiment. The related description of the semi-supervised annotation model training module 60 can refer to the related description of S60 in the above embodiment. The related description of the semi-supervised annotation module 70 can refer to the related description of S70 in the above embodiment.

[0122] In one embodiment, the automatic annotation system 100 of the change detection image further includes an initial model training module, a model output module, a supervised annotation model training module, and a supervised annotation module. The initial model training module is configured to train a supervised annotation model according to the set of repaired object change annotation maps to obtain an initial supervised annotation model. The model output module is configured to input the semi-supervised annotation data set into the initial supervised annotation model to output a set of change detection result maps. The supervised annotation model training module is configured to compare and correct the set of change detection result maps and the set of repaired object change annotation maps, iteratively optimize the initial supervised annotation model, and obtain a trained supervised annotation model. The supervised annotation module is configured to automatically annotate each traffic scene image according to the trained supervised annotation model to generate a supervised annotation data set.

[0123] In this embodiment, the related description of the initial model training module can refer to the related description of S80 in the above embodiment. The related description of the model output module can refer to the related description of S90 in the above embodiment. The related description of the supervised annotation model training module can refer to the related description of S100 in the above embodiment. The related description of the supervised annotation module can refer to the related description of S110 in the above embodiment.

[0124] In an embodiment, the change annotation map extraction module 20 comprises a first change map extraction module, a second change map extraction module, and an AND operation module. The first change map extraction module is configured to perform change region extraction processing on the dual-time-series image set according to a frame difference method to obtain a pixel-level change map. The second change map extraction module is configured to perform change region extraction processing on the dual-time-series image set according to a background difference method to obtain a moving object-level change map. The AND operation module is configured to perform AND operation on the pixel-level change map and the moving object-level change map to obtain the change annotation map.

[0125] In the embodiment, the related description of the first change map extraction module can refer to the related description of S210 in the above embodiment. The related description of the second change map extraction module can refer to the related description of S220 in the above embodiment. The related description of the AND operation module can refer to the related description of S230 in the above embodiment.

[0126] In an embodiment, the object annotation map extraction module 30 comprises a semantic segmentation module, a binary map conversion module, and a mapping module. The semantic segmentation module is configured to segment each dual-time-series image in the dual-time-series image set according to a semantic segmentation model to obtain a corresponding set of attention object mask maps. The binary map conversion module is configured to perform binary map conversion on each set of attention object mask maps to obtain a corresponding set of attention object binary maps. The mapping module is configured to perform pixel-by-pixel or operation mapping of each set of attention object binary maps into the same image to generate the object annotation map.

[0127] In the embodiment, the related description of the semantic segmentation module can refer to the related description of S310 in the above embodiment. The related description of the binary map conversion module can refer to the related description of S320 in the above embodiment. The related description of the mapping module can refer to the related description of S330 in the above embodiment.

[0128] In an embodiment, the repair module 50 comprises a threshold setting module, a noise repair module, and a hole repair module. The threshold setting module is configured to set an independent noise region area threshold and an independent hole region area threshold. The noise repair module is configured to, if the area of an independent noise region in the object change annotation map is less than the independent noise region area threshold, determine that the independent noise region is a target noise region, and remove the independent noise region to obtain a repaired object change annotation map. The hole repair module is configured to, if the area of an independent hole region in the object change annotation map is less than the independent hole region area threshold, determine that the independent hole region is a target noise region, and perform hole filling on the independent hole region to obtain a repaired object change annotation map.

[0129] In the embodiment, the related description of the threshold setting module can refer to the related description of S510 in the above embodiment. The related description of the noise repair module can refer to the related description of S520 in the above embodiment. The related description of the hole repair module can refer to the related description of S530 in the above embodiment.

[0130] In the various embodiments described above, the particular order or hierarchy of steps in processes represented are examples of exemplary approaches. Based on design preferences, it is understood that the particular order or hierarchy of steps in processes represented can be reordered or augmented without departing from the scope of the present disclosure. The accompanying method claims present elements of the various steps in exemplary order and are not intended to be limited to the particular order or hierarchy presented.

[0131] Those of skill would further appreciate that the various illustrative logical blocks, modules, and steps described in connection with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present embodiments.

[0132] The various illustrative logical blocks, and modules described in connection with the embodiments disclosed herein can be implemented or performed with a general purpose processor, a digital signal processor, an application specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor can be a microprocessor, but in the alternative, the general purpose processor can be any conventional processor, controller, microcontroller, or state machine. A processor can also be implemented as a combination of computing devices, e.g., a combination of a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a digital signal processor core, or any other such configuration.

[0133] The steps of the methods or algorithms described in the embodiments of this invention can be directly embedded in hardware, a software module executed by a processor, or a combination of both. The software module can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and storage medium can be housed in an ASIC, which can be housed in a user terminal. Optionally, the processor and storage medium can also be housed in different components of the user terminal.

[0134] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for automated annotation of change detection images, characterized in that The method comprises the following steps: acquire an unsupervised image set, and perform double-phase image division on the unsupervised image set to form a double-phase image set; extract features of pixel changes and moving object changes in the double-phase image set to obtain a plurality of change annotation maps; extract features of focus objects in the double-phase image set to obtain a plurality of object annotation maps; perform AND operation on each of the object annotation maps and each of the change annotation maps to generate a set of object change annotation maps; find a target noise area of each object change annotation map in the set of object change annotation maps, and repair the target noise area to obtain a set of repaired object change annotation maps; acquire a new unlabeled data set, train a semi-supervised annotation model according to the new unlabeled data set and the set of repaired object change annotation maps, and obtain a trained semi-supervised annotation model; automatically annotate double-phase traffic scene images according to the trained semi-supervised annotation model to generate a semi-supervised annotation data set.

2. The method of claim 1, wherein, After the step of automatically annotating double-phase traffic scene images according to the trained semi-supervised annotation model to generate a semi-supervised annotation data set, the method further comprises the following steps: train a supervised annotation model according to the set of repaired object change annotation maps to obtain an initial supervised annotation model; input the semi-supervised annotation data set into the initial supervised annotation model to output a set of change detection result maps; compare and correct the set of change detection result maps and the set of repaired object change annotation maps, iteratively optimize the initial supervised annotation model, and obtain a trained supervised annotation model; automatically annotate each traffic scene image according to the trained supervised annotation model to generate a supervised annotation data set.

3. The method of claim 1, wherein, The step of extracting features of pixel changes and moving object changes in the double-phase image set to obtain a plurality of change annotation maps comprises the following steps: extract a change region of the double-phase image set according to a frame difference method to obtain a pixel-level change map; extract a change region of the double-phase image set according to a background difference method to obtain a moving object-level change map; perform AND operation on the pixel-level change map and the moving object-level change map to obtain the change annotation map.

4. The method of claim 1, wherein, The step of extracting features of focus objects in the double-phase image set to obtain a plurality of object annotation maps comprises the following steps: segment each double-phase image in the double-phase image set according to a semantic segmentation model to obtain a corresponding set of focus object mask maps; convert each set of focus object mask maps into a corresponding set of focus object binary maps; perform pixel-by-pixel OR operation on each set of focus object binary maps to map them into the same image to generate the object annotation map.

5. The method of claim 1, wherein, The step of finding a target noise area of each object change annotation map in the set of object change annotation maps, and repairing the target noise area to obtain a set of repaired object change annotation maps comprises the following steps: set an independent noise area threshold and an independent hole area threshold; If an independent noise point area in the object change annotation map is smaller than the independent noise point area threshold, the independent noise point area is the target noise area, and the independent noise point area is removed to obtain a repaired object change annotation map. Or, if an independent hole area in the object change annotation map is smaller than the independent hole area threshold, the independent hole area is the target noise area, and the independent hole area is filled with holes to obtain the repaired object change annotation map.

6. An automated labeling system of change detection images, characterized in that The system comprises: An image acquisition module is configured to acquire an unsupervised image set, and perform double-phase image division on the unsupervised image set to form a double-phase image set; A change annotation map extraction module is configured to perform feature extraction on pixel changes and moving object changes in the double-phase image set to obtain a plurality of change annotation maps; An object annotation map extraction module is configured to perform feature extraction on a focus object in the double-phase image set to obtain a plurality of object annotation maps; An object change map acquisition module is configured to perform AND operation on each of the object annotation maps and each of the change annotation maps to generate a set of object change annotation maps; A repair module is configured to find a target noise area of each object change annotation map in the set of object change annotation maps, repair the target noise area, and obtain a set of repaired object change annotation maps; A semi-supervised annotation model training module is configured to acquire a new unlabeled data set, train a semi-supervised annotation model according to the new unlabeled data set and the set of repaired object change annotation maps, and obtain a trained semi-supervised annotation model; A semi-supervised annotation module is configured to automatically annotate a double-phase traffic scene image according to the trained semi-supervised annotation model to generate a semi-supervised annotation data set.

7. The automated labeling system of change detection images according to claim 6, wherein, The system further comprises: An initial model training module is configured to train a supervised annotation model according to the set of repaired object change annotation maps to obtain an initial supervised annotation model; A model output module is configured to input the semi-supervised annotation data set into the initial supervised annotation model to output a set of change detection result maps; A supervised annotation model training module is configured to compare and correct the set of change detection result maps and the set of repaired object change annotation maps, iteratively optimize the initial supervised annotation model, and obtain a trained supervised annotation model; A supervised annotation module is configured to automatically annotate each traffic scene image according to the trained supervised annotation model to generate a supervised annotation data set.

8. The automated labeling system of change detection images according to claim 6, wherein, The change annotation map extraction module comprises: A first change map extraction module is configured to perform change area extraction processing on the double-phase image set according to a frame difference method to obtain a pixel-level change map; A second change map extraction module is configured to perform change area extraction processing on the double-phase image set according to a background difference method to obtain a moving object-level change map; An AND operation module is configured to perform AND operation on the pixel-level change map and the moving object-level change map to obtain the change annotation map.

9. The automated labeling system of change detection images according to claim 6, wherein, The object annotation map extraction module comprises: The semantic segmentation module is configured to segment each of the dual-phase images in the dual-phase image set according to a semantic segmentation model to obtain a corresponding set of object-of-interest mask images; The binary image conversion module is configured to convert each of the sets of object-of-interest mask images into a binary image to obtain a corresponding set of object-of-interest binary images; The mapping module is configured to map each of the sets of object-of-interest binary images into a same image by pixel-by-pixel or operation mapping to generate the object annotation image.

10. The automated labeling system of change detection images according to claim 6, wherein, The repair module includes: The threshold setting module is configured to set an independent noise region area threshold and an independent hole region area threshold; The noise repair module is configured to, if an independent noise region area in the object change annotation image is less than the independent noise region area threshold, determine that the independent noise region area is the target noise region, and remove the independent noise region area to obtain a repaired object change annotation image; or The hole repair module is configured to, if an independent hole region area in the object change annotation image is less than the independent hole region area threshold, determine that the independent hole region area is the target noise region, and perform hole filling on the independent hole region area to obtain the repaired object change annotation image.

Images