A remote sensing image change detection and annotation system and method based on global and local combination

Abstract

A remote sensing image change detection and annotation system and method based on global and local combination. Belongs to the field of remote sensing data processing technology. The system includes a task module, a slice production module, a slice annotation module, a data mapping module, a large map annotation module and a display module; the task module creates or opens a remote sensing image change detection and annotation task; the slice production module performs the registration of the annotation data and the slices after registration; the slice annotation module annotates the changes in the slices; the data mapping module realizes the mutual mapping of the slice level and the large map level; the large map annotation module verifies and modifies the changes in the slices; the display module displays the updated large map. The system of the present invention can reduce mislabeling and missing labels and significantly improve the annotation efficiency.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing data processing, and in particular relates to the technical field of remote sensing image change annotation based on global and local combination. Background Art

[0002] With the development of remote sensing technology, change detection has become an important means of obtaining changes in objects in remote sensing images. Change detection mainly compares remote sensing images from different periods to identify the changed areas on the surface. Since this change information has important application value in urban planning, environmental monitoring, agricultural surveys, disaster assessment and other fields, for example, by detecting changes in the surface, it is possible to effectively monitor changes in land use, urban expansion, and the impact of natural disasters. Therefore, in the process of sample annotation, accurate annotation of changed areas becomes crucial. In the process of annotation of change detection sample data, a lot of manpower and time are often required. Usually, the following problems are encountered in the data annotation stage:

[0003] (1) There is a lack of efficient change detection annotation methods and tools. The annotation systems on the market usually only see one phase of the image when annotating, and it is necessary to repeatedly switch images to complete the annotation;

[0004] (2) In the process of annotating samples based on slices, it is impossible to distinguish the categories of small objects at the boundaries;

[0005] (3) In the process of annotating samples based on slices, the annotations of the same changes in different slices are inconsistent;

[0006] (4) In the process of labeling samples based on large images, although the main roads in the image have almost no deviation after registration, there is often a large offset between the front and back objects in local areas. Summary of the invention

[0007] In order to solve the shortcomings of the prior art, the present invention provides a remote sensing image change detection and annotation system based on global and local combination, the system includes a task module, a slice making module, a slice annotation module, a data mapping module, a large map annotation module and a display module;

[0008] The task module creates or opens a remote sensing image change detection and annotation task;

[0009] The slice making module performs registration of the labeled data and slices after registration;

[0010] The slice annotation module annotates changes in the slice;

[0011] The data mapping module realizes the mutual mapping between the slice level and the macro level;

[0012] The large image annotation module verifies and modifies changes in the slices;

[0013] The display module displays the updated large picture.

[0014] Furthermore, the remote sensing image change detection and annotation task includes the following contents: task name, task status, task area, task time, input image path, annotation file path, annotation category and task log.

[0015] Furthermore, in the slice making module, the registration method of the annotation data is:

[0016] S1, using the Fourier-Mellin registration method to register the reference image f1(x, y) and the input image f2(x, y) to obtain a registered image;

[0017] S2. Use a dense residual network to extract features from the registered image, perform log-polar coordinate transformation on the features, apply the Merlin transformation to the transformed features, optimize the rotation and scaling parameters, and solve the transformed features based on the phase correlation technology to obtain the translation parameters, and obtain the preliminary registration results through the dense residual network;

[0018] S3, dividing the preliminary registration results into adaptive grids, establishing a lookup table based on the size of the objects corresponding to the change annotation categories, the resolution of the image, and the image quality information, and jointly determining the size of the adaptive grid;

[0019] S4, dividing the reference image f1(x, y) and the preliminary registration result according to the size of the adaptive grid, to obtain a reference grid image and a preliminary registration grid image;

[0020] S5, registering the reference grid image and the preliminary registration grid image using the Fourier-Mellin registration method to obtain a registered image;

[0021] S7. Use a dense residual network to extract features from the registered image, perform log-polar coordinate transformation on the features, apply the Merlin transform to the transformed features, optimize the rotation and scaling parameters, and solve the transformed features based on the phase correlation technology to obtain the translation parameters, and obtain the fine registration results through a dense residual network.

[0022] Furthermore, in the slice production module, the slices after registration are specifically: the precise registration results are sliced ​​by the set overlap and slice size, and the slices are mosaicked based on pixels rather than geographic locations through the data mapping module to form a mosaic map containing all the slices annotated for the current task - an auxiliary large map, with one slice for each of the two phases, and the positions of each pair of slices in the auxiliary large map are spatially aligned.

[0023] Furthermore, the slice annotation module annotates the changes in the slice specifically as follows:

[0024] Select and confirm the image to be labeled; select the corresponding annotation category according to the change targets to be labeled in the image to be labeled; draw and adjust the annotation polygon based on the range of the change targets to be labeled in the dual-temporal image; identify the objects at the edge of the slice based on the auxiliary large image; combine the dual-temporal slices, delete the mislabeled change targets, draw the missed label change targets, and adjust the mismatched contours; switch the labeled slices to the previous and next ones, and automatically save the current annotations; repeat the above steps until all slices of the current task are labeled; update the task status to preliminary labeling completion.

[0025] Furthermore, the data mapping module is configured to manage the bidirectional mapping of annotated samples from slices to large images; it supports mapping image slices from the slice level to the large image level, it supports mapping annotated samples from the large image level to the slice level; it supports mapping annotation files from the slice level to the large image level.

[0026] Furthermore, the large map annotation module is configured to perform large map annotation tasks; it supports the following functions required for annotating samples: selecting and confirming the image to be annotated; the window it provides supports large-scale image loading and image scaling; it supports repairing inconsistent annotations of the same change in different slices, and for the results that have been annotated in the slices, uses a window overlay to display the auxiliary large map tif and shp, verifies the samples annotated at the slice level, focuses on checking the changes at the edges of the slices, eliminates the changes at the edges of the slices combined with spatial context features, and supports vector editing of the auxiliary large map shp, deletes incorrectly annotated change targets, draws missed annotated change targets, and adjusts non-fitting contours.

[0027] Furthermore, the display module is configured to display the labeled samples and display the colors corresponding to the different category changes in the legend part.

[0028] The present invention also provides a remote sensing image change detection and annotation method based on global and local combination, the method is implemented using the remote sensing image change detection and annotation system based on global and local combination as described in any one of the above items, and the method comprises the following steps:

[0029] S1. Use the task module to create a change detection annotation task;

[0030] S2, using the image specified in step S1, through the slice making module, performing registration of the annotation data and preprocessing of the slices;

[0031] S3, using a slice annotation module to annotate changes in the slices obtained in step S2;

[0032] S4, using the data mapping module, mapping the annotated samples from the slice level to the macro level;

[0033] S5. Use the large-image annotation module to verify and modify annotation changes;

[0034] S6. Use the data mapping module to map the labeled samples from the large image to the slices, and update the task status to completed;

[0035] S7. Use the task module to manage the exported slice samples.

[0036] The beneficial effects of the present invention are:

[0037] (1) The present invention provides a remote sensing image change detection and annotation system based on global and local integration, and the data mapping module it covers supports the bidirectional conversion of annotation samples from the slice level to the auxiliary map level. This enables the method to distinguish the changes of objects from the spatial context of the dual-temporal image slices and the larger range of the area to be annotated, and use it in the sample annotation process of change detection. It takes into account the fact that the dual-temporal images may contain the contours of the objects to be drawn, the objects in adjacent slices and the incomplete objects at the edges of the slices, etc. This can reduce mislabeling and missing labels, improve the performance of the system, and significantly improve the annotation efficiency.

[0038] (2) The present invention provides a remote sensing image change detection and annotation system based on the combination of global and local, which uses a registration method that is an improved quadratic Fourier-Mellin registration method using a dense residual network. For each pair of slices in the task area, the quadratic registration provided by this method can ensure that the same objects at different times are highly aligned in space at both the global and local scales, thereby avoiding the situation in which, in traditional change detection and annotation, only one full-image registration is performed, resulting in almost no deviation of the main roads in the image after geometric correction, but a large offset of the front and rear objects often occurs in the local area.

[0039] (3) The remote sensing image change detection and annotation system provided by the present invention, which is based on the combination of global and local, can solve the problem of inconsistency across slices when annotating samples based on overlapping slices while retaining high registration accuracy. It performs original annotation based on slices with partial overlap, retains the high spatial consistency of the objects in the dual-phase slices, and combines the data mapping module and the large-scale annotation module to verify the missed areas. When annotating samples based on overlapping slices, it can ensure that the same changes in multiple slices are labeled consistently, prevent the same changes from being labeled only in some slices, and increase the internal consistency of the training samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a structural diagram of the remote sensing image change detection and annotation system based on global and local combination according to the present invention;

[0041] Figure 2A schematic diagram of a remote sensing image change detection and annotation method based on global and local combination according to the present invention;

[0042] Figure 3 Schematic diagram of the improved quadratic Fourier-Mellin registration method for dense residual networks;

[0043] Figure 4 A schematic diagram of an auxiliary large image tif generated by the remote sensing image change detection and annotation method based on global and local combination of the present invention;

[0044] Figure 5 A schematic diagram of the superimposed display of the auxiliary large image tif and the auxiliary large image shp generated by the remote sensing image change detection and annotation method based on the global and local combination of the present invention;

[0045] Figure 6 It is a schematic diagram of an enlarged window of a slice annotation module of the remote sensing image change detection and annotation system based on global and local combination of the present invention;

[0046] Figure 7 It is a window reduction schematic diagram of a slice annotation module of the remote sensing image change detection and annotation system based on global and local combination of the present invention;

[0047] Figure 8 It is a schematic diagram of the annotation results of the remote sensing image change detection and annotation system based on global and local combination of the present invention;

[0048] Fig. 9 The present invention is a schematic diagram of the data annotation process of the remote sensing image change detection and annotation system based on global and local combination. DETAILED DESCRIPTION

[0049] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0050] Embodiment 1,

[0051] This embodiment provides a remote sensing image change detection and annotation system based on global and local combination, which solves the shortcomings of the existing technology based only on large map annotation or only on slice annotation. The existing change detection and annotation systems are not mature enough. In the process of annotating samples based on slices, the categories of small objects at the border cannot be distinguished, and the annotations of the same change in different slices are inconsistent; in the process of annotating samples based on large maps, although the main roads in the image have almost no deviation after registration, there is often a large offset between the front and back objects in the local area. The existing methods are relatively simple in the preprocessing of the original image during the annotation process, and do not fully utilize the change information contained in the dual-phase image slices and the larger range of spatial context of the area to be annotated during the annotation process.

[0052] The system structure diagram is as follows Figure 1 As shown, the system includes a task module, a slice making module, a slice marking module, a data mapping module, a large map marking module and a display module;

[0053] The task module creates or opens a remote sensing image change detection and annotation task;

[0054] The slice making module performs registration of the labeled data and slices after registration;

[0055] The slice annotation module annotates changes in the slice;

[0056] The data mapping module realizes the mutual mapping between the slice level and the macro level;

[0057] The large image annotation module verifies and modifies changes in the slices;

[0058] The display module displays the updated large picture.

[0059] Each module is introduced in detail.

[0060] The task module is configured to create and manage the task information of the image to be annotated and manage the sample data. The operator can complete the task creation by entering the following attributes: task name, task status, task area, task time, input image path, annotation file path, and annotation category. The task module can view the task status and task log of all current change detection and annotation tasks, and through the associated functional modules, it can select and execute the specified task. It can find the most suitable data for the scene from the existing data set based on the task characteristics of the specified scene, and form the data set that best suits the current scene.

[0061] The slice production module is configured to manage the preprocessing of image annotation data, such as registration and slicing. The preprocessing includes the quadratic Fourier-Mellin registration method improved by dense residual network and the data cut into small slices with specified overlap. For each pair of slices in the task coverage area, two registrations can ensure that the same features at different times are highly aligned in space at both the overall and local scales. Taking 0.8m high-resolution image as an example, the mean square error of the existing registration algorithm after registration is less than 3m, but in many areas the house deviation can reach 5m, which is equivalent to 7 pixels. The high-precision change detection and annotation process requires the deviation of ground objects to be within 3 pixels. The two-time registration method proposed in this method can control the deviation of ground objects in all slices within 3 pixels, which can significantly improve the quality of labeled samples and allow the change detection model to learn information related to ground object changes; then use the data mapping module to mosaic the slices based on the row and column numbers of the slices and the pixel coordinates to form a slice mosaic. In this image, there are black borders with an interval of 30 pixels between slices in different columns and rows, and each slice has a row and column number. The relative position between different slices is the same as the actual situation. The slices of the previous and next phases are processed separately to generate two auxiliary large images tif.

[0062] The slice annotation module is configured to perform slice annotation tasks, and it supports the following functions required for annotation samples. Select and confirm the image to be annotated; select the corresponding annotation category according to the change target to be annotated in the image to be annotated, and use different colors to distinguish the changes of different categories in the annotation; the annotation interface it provides includes two windows, which contain the current slices of the two phases respectively. The two windows support synchronous zooming and moving, which helps the operator to quickly interpret and mark the changed objects even when the slices of the previous and next phases contain the outline of the changed objects. Use the mouse brush function to draw the annotation polygon in one window, and the annotation polygon and brush will be displayed synchronously in another window, such as Figure 6 and Figure 7 As shown in the figure, the annotation polygon is drawn and adjusted in combination with the range of the change target to be annotated in the dual-phase image; for the objects at the edge of the slice, click the auxiliary large image view button to view the spatial context information of the slice. In addition, the slice annotation module also supports the auditor's rapid review. Combined with the dual-phase window of the module, the wrongly annotated change targets can be deleted, the missing annotated change targets can be drawn, and the non-fitting contours can be adjusted; the annotated slice can be switched to the previous or next one, the current annotation can be automatically saved, and the new slice and its annotation can be loaded.

[0063] The large map annotation module is configured to perform large map annotation tasks; it supports the following functions required for annotating samples: selecting and confirming the image to be annotated; the window it provides supports large-scale image loading and image scaling; it supports repairing inconsistent annotations of the same change in different slices, and for the results that have been annotated in the slice, uses a window to overlay the auxiliary large map tif and shp, verifies the samples annotated at the slice level, focuses on verifying the changes at the edges of the slices, eliminates the changes at the edges of the slices combined with spatial context features, and supports vector editing of the auxiliary large map shp, deletes mis-annotated change targets, draws missed change targets, and adjusts non-fitting contours.

[0064] The data mapping module is configured to manage the bidirectional mapping of annotated samples from slices to large images. It supports mapping image slices from the slice level to the large image level. The generated auxiliary large image tif file contains the ground feature information of all slices in the original image, and can also ensure that all slice pairs used for change detection annotation are mutually registered. All slices can be quickly viewed using a window that only loads one image. It supports mapping annotated samples from the large image level to the slice level. It supports mapping annotation files from the slice level to the large image level. Specifically, the json files obtained by slice annotation are mosaicked based on pixel coordinates to form an auxiliary large image shp file that corresponds to the auxiliary large image tif file in space. The file covers all the annotation information of the current task. It supports mapping annotation files from the large image level to the slice level, which is convenient for quickly restoring the review results of the large image annotation module to samples in the form of slices that can be used for training. This module builds a bridge between the two scales of slices and large images, making it possible to deeply couple the information of the two scales for annotation.

[0065] The display module is configured to display the labeled samples. The legend part shows the colors corresponding to different types of changes, which is convenient for researchers to quickly label according to the frequency of occurrence of different types of changes in the area.

[0066] In the task module, when managing and producing data sets, the distribution of expected categories can be manually input. The annotation system will calculate the task characteristics based on the expected categories and display the tasks with the closest task characteristic values ​​in the task library. When the expected tasks are selected when producing data sets, the sample set required for this change detection task will be output in the specified folder, and a supporting database file will be generated to store the original image of each slice, the row and column numbers of the corresponding slices, the annotation categories and other information. Combined with the task log, the full traceability of the slices can be achieved.

[0067] A similar task recommendation method combined with F1 weighted analysis is proposed: based on the category of ground feature changes and the estimation accuracy of different change categories, the existing multi-dimensional information is reduced to m-dimensional based on the F1-weighted main feature extraction method, and the tasks whose similarity with the specified task is greater than the task characteristic similarity threshold in the m-dimensional space are found, and three recommended multi-dataset combination ratios are given.

[0068] The F1-weighted main feature extraction method is a technique for dimensionality reduction of multidimensional data. It aims to extract the main features by weighted combination of F1-scores and reduce the data dimension while retaining as much useful information as possible. F1-score is an important indicator of classification model performance, combining precision and recall:

[0069]

[0070] In the feature extraction process, F1-score is used to measure the importance of different features in a specific task, and then used to weight these features to extract the main features that best represent the original data.

[0071] Feature selection and F1 weighting:

[0072] For each feature, calculate its F1-score in a specific task. The higher the F1-score, the more important the feature is to the task.

[0073] According to the value of F1-score, each feature is weighted. Assume that the original data matrix is ​​X, which represents the proportion of samples of different categories, and the matrix after feature weighting is X ′ , which can be expressed as:

[0074] X ′ =X×W;

[0075] Among them, W is a diagonal matrix composed of the F1-score of each feature.

[0076] Principal Component Analysis (PCA):

[0077] For the weighted data matrix X ′ Perform principal component analysis to extract the main features (principal components). The basic formula of PCA is:

[0078] Y=X ′ ×V;

[0079] Among them, V is the matrix composed of eigenvectors obtained by the PCA method, and Y is the data matrix after dimensionality reduction.

[0080] Determine the dimensions after dimensionality reduction:

[0081] The dimension m after dimensionality reduction is determined by analyzing the cumulative variance contribution rate. Usually, the number of dimensions is selected so that the cumulative variance contribution rate reaches a preset threshold (such as 95%).

[0082] Similarity-based task recommendation:

[0083] After reducing the dimension to m-dimensional, for new tasks, we can calculate their cosine similarity with existing tasks in m-dimensional space to find tasks with similarity greater than a certain threshold. The cosine similarity formula is as follows:

[0084]

[0085] Among them, Y1 and Y2 are the feature vectors of the two tasks in the m-dimensional space.

[0086] Recommended combination ratio of multiple datasets:

[0087] Based on the features extracted by the above method, three different ratios of multi-dataset combinations are recommended, the difference lies in the similarity between different tasks, including:

[0088] High-similarity task combination: Select the dataset with the highest similarity of task feature vectors, and the proportion is biased towards this dataset.

[0089] Moderately similar task combination: Select multiple datasets with moderate similarity and a balanced ratio.

[0090] Low-similarity task combination: Select datasets with low similarity but diverse characteristics. The proportion can be adjusted according to specific needs.

[0091] Through the F1-weighted main feature extraction method, key information can be retained while reducing the dimension, and task recommendation can be achieved by combining cosine similarity. This method not only effectively reduces the data dimension, but also enhances the similarity analysis between different tasks, providing a theoretical basis for the reasonable combination of multiple data sets.

[0092] like Fig. 9 The figure shows a schematic diagram of the data annotation process of the remote sensing image change detection and annotation system based on global and local combination disclosed in Example 1.

[0093] The remote sensing image change detection and annotation system based on global and local combination disclosed in Example 1 includes a data annotation process, which specifically includes:

[0094] Use the task module to create a change detection annotation task, and enter the following parameters: task area, task time, input image path, annotation file path, annotation category, and task log;

[0095] Through the slice making module, the preprocessing of the annotated data such as registration and slicing is performed, and the preprocessing includes the dense residual network improved quadratic Fourier-Mellin registration method of the original image, cropping, and mosaicking into a slice mosaic based on pixel coordinates;

[0096] Use the slice annotation module to quickly annotate and modify changes in slices;

[0097] Use the data mapping module to map the annotated samples from the slice level to the map level. Specifically, the json file obtained by slice annotation is mosaicked based on pixel coordinates to form an auxiliary map shp file that corresponds to the auxiliary map tif file in space. This file covers all the annotated information of the current task.

[0098] Use the large-scale annotation module to quickly verify and modify annotation changes;

[0099] Call the data mapping module to map the labeled samples from the large image to the slices;

[0100] The task module manages and exports slice samples and performs change detection.

[0101] Embodiment 2,

[0102] This embodiment is based on the embodiment 1 and provides a remote sensing image change detection and annotation method based on global and local combination. Figure 2 The specific steps are as follows:

[0103] Step 1: Use the task module to create or open a change detection annotation task. The task contains the following properties: task name, task status, task area, task time, input image path, annotation file path, annotation category, task log, and task characteristics. Task characteristics represent the proportion of annotation change categories. Task characteristics and task log are empty when a task is created. As the annotation progresses, task characteristics, task status, and task log will be continuously updated.

[0104] Step 2: Using the image specified in step 1, the slice making module is used to perform preprocessing such as registration and slicing of the annotated data. The preprocessing includes the dense residual network improved quadratic Fourier-Mellin registration of the original image and cropping to small slice data with a specified overlap. For each pair of slices in the task coverage area, two registrations can ensure that the same objects at different times are highly aligned in space at both the overall and local scales. Then, the data mapping module is used to mosaic the slice mosaics based on the row and column numbers of the slices and the pixel coordinates, which are referred to as "auxiliary large image tif" in this patent, such as Figure 4As shown, in this image, there are black borders with an interval of 30 pixels between slices in different columns and rows. There are row and column numbers under each slice. The relative positions between different slices are the same as the actual situation. The slices of the previous and next phases are processed separately to generate two auxiliary large images tif.

[0105] Step 3: Use the slice annotation module to quickly annotate the changes in the slices obtained in step 2. The annotation includes: selecting and confirming the image to be annotated; selecting the corresponding annotation category according to the change target to be annotated in the image to be annotated; drawing and adjusting the annotation polygon based on the range of the change target to be annotated in the dual-phase image; identifying the ground objects at the edge of the slice based on the auxiliary large image tif; combining the dual-phase slices, deleting the mis-annotated change targets, drawing the missed change targets, and adjusting the mismatched contours; switching the annotated slices to the previous and next slices, and automatically saving the current annotation; repeating the above steps until all slices of the current task are annotated; updating the task status to preliminary annotation completion.

[0106] Step 4: Use the data mapping module to map the labeled samples from the slice level to the large image level. Specifically, the json file obtained by the slice annotation in step 3 is mosaicked based on the pixel coordinates to form an auxiliary large image shp file that corresponds to the auxiliary large image tif file in space. This file covers all the labeled information of the current task.

[0107] Step 5: Use the large map annotation module to quickly verify and modify the annotation changes. The process includes: selecting and confirming the image to be annotated; repairing the inconsistency of annotations for the same change in different slices; combining the spatial context objects, deleting the mis-annotated change targets, drawing the missed annotation change targets, and adjusting the non-fitting contours. Specifically, for the results processed in step 4, use a single window that overlays the auxiliary large map tif and shp, such as Figure 5 As shown, the samples annotated at the slice level are verified, with a focus on checking the changes at the slice edges to eliminate the problem of inconsistent annotations at the slice edges. Then, the task status map is updated and the verification is completed.

[0108] Depending on the importance of the task and the different mapping experience of the participants, iteratively repeat steps 3, 4, and 5 according to the specific situation until there are basically no missed or mislabeled samples.

[0109] Step 6: Call the data mapping module to map the labeled samples from the large image to the slices, update the task status to complete, use the task module to manage and export the slice samples, and perform change detection. The dataset generated by this method is as follows Figure 8 As shown. According to the task requirements specified by the user, a similar task recommendation method combined with F1 weighted analysis is proposed, and the best recommended combination of three existing data sets is given to generate a data set.

[0110] Embodiment 3,

[0111] This embodiment is a further limitation of Embodiment 2.

[0112] In step 1, the task characteristics represent the proportion of the annotation change category. The task characteristics and task log are both empty when the task is newly created. As the annotation progresses, the task characteristics, task status and task log are continuously updated.

[0113] In step 2, the registration method used is the quadratic Fourier-Mellin registration method improved by the dense residual network, and its process is as follows: Figure 3 shown.

[0114] Assume that the input image f2(x,y) is the reference image f1(x,y) after translation (x0, y0) and rotation Scaling the image obtained after σ transformation, we have:

[0115]

[0116] Take the Fourier transform:

[0117] Perform Fourier transform on the input image f2(x,y) and the reference image f1(x,y) respectively to obtain a spectrum diagram.

[0118] For the reference image f1(x,y) and the input image f2(x,y), their spectrograms obtained after Fourier transform are F1(u,v) and F2(u,v), respectively. Modulo operation is performed on F1(u,v) and F2(u,v), respectively. After the modulo operation, the power relationship between the two satisfies:

[0119]

[0120] Perform a log-polar transformation:

[0121] Convert the spectrum to log polar coordinates to more easily handle the effects of rotation and scaling. From the formula obtained after the modulo operation, it can be seen that the spectrum amplitude is only related to the rotation and scaling factors, and has nothing to do with the translation parameters. Ignore the scaling factor in the formula and perform a log polar conversion on the amplitude value:

[0122] M2(logρ,θ)=M1(logρ-logσ,θ-θ0); where, ρ and θ are polar coordinates, M2 and M1 represent the images after log polar conversion;

[0123] Phase correlation method:

[0124] According to the phase correlation method, the rotation and scaling parameters can be obtained. Then the image to be registered is restored by rotation and scaling transformation. The restored image has only a translation transformation with the reference image, and the frequency domain phase correlation method is used again to obtain the translation variable.

[0125] Angle, scale, translation compensation:

[0126] The rotation and scaling factors are determined by the phase correlation extreme points, and the translation is determined by calculating the phase difference caused by the translation. According to the estimated rotation, scaling and translation parameters, the image to be registered is geometrically transformed accordingly to align it with the reference image.

[0127] The Fourier-Mellin method performs well in image registration with rotation, scaling, and translation, and has a certain degree of noise resistance.

[0128] Dense Residual Networks:

[0129] The dense residual network mainly improves the transmission and reuse of features by establishing denser connections between layers, thereby enhancing the learning ability of the network. Combined with the Fourier-Mellin method, the dense residual network can be used to extract and match image features more efficiently.

[0130] The input image, i.e., the image after registration, is subjected to feature extraction, and a dense residual network is used to extract the features of the input image, replacing the direct Fourier transform in the traditional Fourier-Mellin method. This method can effectively prevent the gradient from disappearing and improve the efficiency and accuracy of feature extraction through the combination of residual connection and dense connection. The specific operation is as follows: log polar coordinate transformation of the features, application of the Merlin transform to the transformed features, and then optimization to obtain the final registration parameters: the features extracted by the dense residual network are combined with the Merlin transform to process rotation and scaling, and the registration parameters obtained by optimization include scaling parameters and rotation parameters, and the translation parameters can be obtained through phase correlation technology.

[0131] Adaptive mesh parameter calculation:

[0132] The preliminary registration results obtained by dense residual network registration can ensure the alignment of the main features in the image, such as main roads and factory areas, and the mean error of feature offset is within 5m. However, in order to ensure the height alignment of the features in all slices, it is necessary to determine the adaptive grid parameters and divide the dense preliminary registration results into geographic grids. Based on the size of the features corresponding to the change annotation category, the resolution of the image, the image quality and other information, a lookup table is established to jointly determine the size of the adaptive grid. When the image quality is A-level and the feature size is 10m, setting the adaptive grid parameter to 758 can effectively ensure high-precision registration of the features in the slice. In the process of calculating the adaptive grid parameters, the grid misalignment caused by the registration is alleviated by adding overlapping areas, which can ensure that after registration, there will be no black edges in the final labeled slice.

[0133] Grid registration:

[0134] Dense residual networks are applied at the grid level to improve the transmission and reuse of features and enhance the learning ability of the network by establishing denser connections between layers. Combined with the Fourier-Mellin method, the dense residual network can further reduce errors in angles, scales, etc., and can be used to more efficiently extract and match small-scale image features.

[0135] Other follow-up processing:

[0136] Based on the finely registered slice pairs obtained by the residual quadratic registration method, the slices are mosaicked based on pixels rather than geographic locations through the data mapping module, forming a mosaic map containing all the annotated slices of the current task - the auxiliary large map, with one for each of the previous and next phases. The positions of each pair of slices in the auxiliary large map tif are highly aligned in space.

[0137] In step 3, the annotation interface provided includes two windows, each containing the current slices of the two phases. The two windows support synchronous scaling and movement, which helps the operator to quickly interpret and mark the changing objects even when the slices of the previous and subsequent phases contain the outlines of the changing objects. When drawing polygons, the mouse brush will appear in the two windows at the same time, move synchronously and draw polygons.

[0138] In step 5, the auxiliary large image used is a mosaic image containing all the slices annotated for the current task, with one image each for the two phases before and after. Roll-up display and layer overlay display can be used, and the positions of each pair of slices in the auxiliary large image tif are highly aligned in space.

Claims

1. A remote sensing image change detection and annotation system based on global and local combination, characterized in that: The system includes a task module, a slice making module, a slice annotation module, a data mapping module, a large map annotation module and a display module; The task module creates or opens a remote sensing image change detection and annotation task; The slice making module performs registration of the labeled data and slices after registration; The slice annotation module annotates changes in the slice; The data mapping module realizes the mutual mapping between the slice level and the macro level; The large image annotation module verifies and modifies changes in the slices; The display module displays the updated large picture.

2. The remote sensing image change detection and annotation system based on global and local combination according to claim 1 is characterized in that: The remote sensing image change detection and annotation task includes the following contents: task name, task status, task area, task time, input image path, annotation file path, annotation category and task log.

3. The remote sensing image change detection and annotation system based on global and local combination according to claim 2 is characterized in that: In the slice making module, the registration method of the annotation data is: S1, using the Fourier-Mellin registration method to register the reference image f1(x, y) and the input image f2(x, y) to obtain a registered image; S2. Use a dense residual network to extract features from the registered image, perform log-polar coordinate transformation on the features, apply the Merlin transformation to the transformed features, optimize the rotation and scaling parameters, and solve the transformed features based on the phase correlation technology to obtain the translation parameters, and obtain the preliminary registration results through the dense residual network; S3, dividing the preliminary registration results into adaptive grids, establishing a lookup table based on the size of the objects corresponding to the change annotation categories, the resolution of the image, and the image quality information, and jointly determining the size of the adaptive grid; S4, dividing the reference image f1(x, y) and the preliminary registration result according to the size of the adaptive grid, to obtain a reference grid image and a preliminary registration grid image; S5, registering the reference grid image and the preliminary registration grid image using the Fourier-Mellin registration method to obtain a registered image; S7. Use a dense residual network to extract features from the registered image, perform log-polar coordinate transformation on the features, apply the Merlin transform to the transformed features, optimize the rotation and scaling parameters, and solve the transformed features based on the phase correlation technology to obtain the translation parameters, and obtain the fine registration results through a dense residual network.

4. The remote sensing image change detection and annotation system based on global and local combination according to claim 3 is characterized in that: In the slice production module, the registered slices are specifically: the precise registration results are sliced ​​by the set overlap and slice size, and the slices are mosaicked based on pixels rather than geographic locations through the data mapping module to form a mosaic map containing all the annotated slices of the current task - the auxiliary large map, with one slice for each of the two phases. The positions of each pair of slices in the auxiliary large map are spatially aligned.

5. The remote sensing image change detection and annotation system based on global and local combination according to claim 4 is characterized in that: The slice annotation module annotates the changes in the slice specifically as follows: Select and confirm the image to be labeled; select the corresponding annotation category according to the change targets to be labeled in the image to be labeled; draw and adjust the annotation polygon based on the range of the change targets to be labeled in the dual-temporal image; identify the objects at the edge of the slice based on the auxiliary large image; combine the dual-temporal slices, delete the mislabeled change targets, draw the missed label change targets, and adjust the mismatched contours; switch the labeled slices to the previous and next ones, and automatically save the current annotations; repeat the above steps until all slices of the current task are labeled; update the task status to preliminary labeling completion.

6. The remote sensing image change detection and annotation system based on global and local combination according to claim 5 is characterized in that: The data mapping module is configured to manage the bidirectional mapping of the labeled samples from the slice to the large image; it supports mapping the image slices from the slice level to the large image level, and it supports mapping the labeled samples from the large image level to the slice level; It supports mapping annotation files from the slice level to the macro level.

7. The remote sensing image change detection and annotation system based on global and local combination according to claim 6 is characterized in that: The large-image annotation module is configured to perform large-image annotation tasks; It supports the following functions required for labeling samples: selecting and confirming the images to be labeled; the window it provides supports large-scale image loading and image zooming; It supports repairing inconsistent annotations of the same change in different slices. For the results that have been annotated in the slices, it uses a window overlay to display the auxiliary large-scale tif and shp, verifies the samples annotated at the slice level, and focuses on verifying the changes at the edges of the slices, eliminating the changes at the edges of the slices combined with the spatial context features. It supports vector editing of the auxiliary large-scale shp, deletes incorrectly annotated change targets, draws missed annotated change targets, and adjusts non-fitting contours.

8. The remote sensing image change detection and annotation system based on global and local combination according to claim 7 is characterized in that: The display module is configured to display the labeled samples and display the colors corresponding to the different category changes in the legend part.

9. A remote sensing image change detection and annotation method based on global and local combination, characterized in that: The method is implemented using the remote sensing image change detection and annotation system based on global and local combination as described in any one of claims 1 to 8, and the method comprises the following steps: S81, using the task module to create a change detection annotation task; S82, using the image specified in step S81, through the slice making module, performing registration of the annotation data and preprocessing of the slices; S83, using a slice annotation module to annotate changes in the slices obtained in step S82; S84, using a data mapping module, mapping the annotated samples from the slice level to the macro level; S85. Use the large-image annotation module to verify and modify annotation changes; S86. Use the data mapping module to map the labeled samples from the large image to the slices, and update the task status to completed. S87. Use the task module to manage the exported slice samples.

Images