Ground feature change detection method and device

Abstract

The invention discloses a ground object change detection method and device, and relates to the technical field of image processing, and the detection method comprises the steps: collecting image data of a target area, employing a central processor and a graphics processor, carrying out the parallel registration processing of the image data, obtaining aligned image data, and storing the aligned image data in a database; and inputting the aligned image data into a preset ground object change detection model, and outputting a detection result. According to the invention, the technical problem of low timeliness of ground feature change detection caused by low efficiency of image registration in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method and device for detecting ground object changes. Background Art

[0002] With the development of computer technology and remote sensing technology, the spatial resolution and temporal resolution of remote sensing images have been greatly improved, providing richer information for image analysis using remote sensing images and time series. In addition, remote sensing images can provide high-precision and high-frequency reconnaissance data, making it possible to perform change analysis based on multi-temporal data. As an important application in remote sensing image analysis, change detection refers to the use of remote sensing images to analyze the change information contained in objects and targets obtained at the same geographical location at different times.

[0003] Currently, remote sensing technology and computer vision and other technical means can be used to analyze the shape, texture, color and other features in remote sensing images, distinguish between background and target areas, and extract useful ground information from them, thereby serving applications in traffic detection, environmental monitoring, ocean detection, terrain surveying and mapping, and resource exploration.

[0004] In related technologies, deep neural networks can be used for change detection. The change detection process is as follows: first, the input remote sensing image is preprocessed; then, horizontal and vertical features are extracted from the remote sensing image; then, the extracted features are processed and fused; finally, the fused features are post-processed to obtain the final change detection result.

[0005] Currently, the prerequisite for using deep neural networks for change detection is to ensure that the two images under test are spatially consistent, that is, image registration. Low image registration accuracy will lead to a significant reduction in the performance of subsequent target and region change detection, and even the subsequent change detection task will not be able to proceed normally due to registration failure.

[0006] However, the current image registration algorithm is very time-consuming, which can easily lead to the overall change detection time failing to meet the real-time requirements. At the same time, the current change detection algorithm may easily detect large areas of ground object changes in the image, but due to the interference of irregular and complex scenes in the image, the edge information of the ground object area may be lost due to the downsampling and upsampling process of the deep network, resulting in low detection accuracy.

[0007] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0008] The embodiments of the present invention provide a method and device for detecting changes in ground objects, so as to at least solve the technical problem in the related art that the efficiency of image registration is low, resulting in low timeliness in detecting changes in ground objects.

[0009] According to one aspect of an embodiment of the present invention, a method for detecting changes in ground objects is provided, comprising: collecting image data of a target area, wherein the image data comprises: a plurality of images of the target area taken at different time points; using a central processing unit and a graphics processing unit to perform parallel registration processing on the image data to obtain aligned image data, wherein the aligned image data comprises: a plurality of aligned images aligned in spatial coordinates; inputting the aligned image data into a preset ground object change detection model, and outputting a detection result, wherein the preset ground object change detection model comprises at least: an attention structure and a pixel enhancement structure, and the detection result comprises at least: change images sorted in chronological order, each change image being annotated with a change sub-area.

[0010] Furthermore, a central processing unit and a graphics processing unit are used to perform parallel registration processing on the image data to obtain the steps of aligned image data, including: storing the image data in the central processing unit, preprocessing the image data on the central processing unit to obtain target image data, and transmitting the target image data to the graphics processing unit; processing the target image data by the central processing unit and the graphics processing unit to obtain a set of candidate extreme points, wherein each candidate extreme point in the set of candidate extreme points corresponds to point information, and the point information at least includes: the location of the candidate extreme point and the resolution scale corresponding to the candidate extreme point; screening the set of candidate extreme points by the central processing unit and the graphics processing unit to obtain a set of target extreme points, wherein each target extreme point in the set of target extreme points corresponds to target point information; based on the set of target extreme points, aligning each target image in the target image data by the central processing unit and the graphics processing unit to obtain aligned image data.

[0011] Furthermore, the step of using a central processing unit and a graphics processing unit to process the target image data to obtain a set of candidate extreme value points includes: performing Gaussian filtering on the target image data on the graphics processing unit to obtain a pyramid image, and transmitting the pyramid image to the central processing unit, wherein the pyramid image includes: multiple layers of images, and the resolution scale of each layer of the image is different; performing differential processing on the pyramid image on the central processing unit to obtain a differential pyramid image, wherein the differential pyramid image includes: multiple layers of differential images, and each layer of differential image is an image obtained by differentially performing two adjacent layers of images in the pyramid image; for each layer of differential image, determining candidate extreme value points on the differential image based on the upper and lower layer differential images adjacent to the differential image on the central processing unit; determining the position and resolution scale of each candidate extreme value point, obtaining point information of each candidate extreme value point, and adding all candidate extreme value points to the set of candidate extreme value points.

[0012] Furthermore, a central processing unit and a graphics processing unit are used to screen a set of candidate extreme points to obtain a step of a target extreme point set, including: transmitting the set of candidate extreme points to the graphics processing unit, and updating the position of each candidate extreme point on the graphics processing unit based on the point information corresponding to each candidate extreme point; determining the second-order derivative matrix of each candidate extreme point on the graphics processing unit, and determining the principal curvature of each candidate extreme point based on the second-order derivative matrix; and screening the target extreme points indicated by the principal curvature less than a preset principal curvature threshold from the extreme point set on the graphics processing unit to obtain the target extreme point set.

[0013] Furthermore, based on the target extreme point set, each target image in the target image data is aligned using a central processing unit and a graphics processing unit to obtain the steps of aligned image data, including: transmitting the target extreme point set to the central processing unit, and determining the target direction of each target extreme point on the central processing unit based on the target point information corresponding to each target extreme point; determining the feature descriptor of each target extreme point on the central processing unit based on the target direction of each target extreme point; transmitting the feature descriptor of each target extreme point to the graphics processing unit, and aligning each target image in the target image data on the graphics processing unit based on the feature descriptor of each target extreme point to obtain aligned image data.

[0014] Furthermore, the step of aligning each target image in the target image data based on the feature descriptor of each target extreme point on the graphics processor to obtain aligned image data includes: determining the similarity value between every two target extreme points based on the feature descriptor; for each target extreme point, determining another target extreme point whose similarity value with the target extreme point is the largest, and determining the target extreme point and the other target extreme point as a matching point pair; and aligning each target image in the target image data based on all matching point pairs to obtain aligned image data.

[0015] Furthermore, before the aligned image data is input into the preset land feature change detection model and the detection result is output, it also includes: adding an attention structure and a pixel enhancement structure to the initial network structure to obtain a preset network structure, wherein the preset network structure also includes: an encoder and a decoder; the attention structure is used to perform weighted processing on each input aligned image to obtain a weighted feature map; the pixel enhancement structure is used to perform pixel enhancement processing on each input weighted feature map; collecting historical image data, wherein the historical image data includes: a set of historical area images taken at multiple time points in different areas; annotating all historical area images in each historical area image set to obtain historical image annotation data; freezing the weight parameters of the encoder in the preset network structure, and using the historical image annotation data to iteratively train the weight parameters of the decoder in the preset network structure to obtain a preset land feature change detection model.

[0016] Furthermore, the attention structure performs weighted processing on each aligned image input to obtain a weighted feature map, including: extracting features from each aligned image to obtain an original feature map of each aligned image; determining the weight of each feature point on the original feature map to obtain a weighted feature map; obtaining an attention weighted feature map based on the attention weight matrix and the weighted feature map; and fusing the attention weighted feature map with the original feature map to obtain a weighted feature map.

[0017] Furthermore, the pixel enhancement structure performs pixel enhancement processing on each input weighted feature map, including: decomposing each weighted feature map to obtain a superpixel set, wherein the superpixel set includes: multiple superpixels, each superpixel is a region composed of a group of pixels with adjacent positions and similar features; performing feature extraction on each superpixel to obtain pixel feature information of each superpixel; comparing each pixel feature information of adjacent weighted feature maps in chronological order to obtain a change block; merging all change blocks to obtain a change region, wherein the change region is used to generate a change image.

[0018] According to another aspect of an embodiment of the present invention, a device for detecting changes in ground objects is also provided, including: an acquisition unit, used to acquire image data of a target area, wherein the image data includes: a plurality of images of the target area taken at different time points; a processing unit, used to perform parallel registration processing on the image data using a central processing unit and a graphics processing unit to obtain aligned image data, wherein the aligned image data includes: a plurality of aligned images aligned in spatial coordinates; an output unit, used to input the aligned image data into a preset ground object change detection model, and output a detection result, wherein the preset ground object change detection model includes at least: an attention structure and a pixel enhancement structure, and the detection result includes at least: change images sorted in chronological order, and each change image is annotated with a change sub-area.

[0019] Furthermore, the processing unit includes: a first processing module, which is used to store the image data in a central processing unit, and pre-process the image data on the central processing unit to obtain target image data, and transmit the target image data to a graphics processing unit; a second processing module, which is used to use the central processing unit and the graphics processing unit to process the target image data to obtain a set of candidate extreme points, wherein each candidate extreme point in the candidate extreme point set corresponds to point information, and the point information at least includes: the location of the candidate extreme point and the resolution scale corresponding to the candidate extreme point; a first screening module, which is used to use the central processing unit and the graphics processing unit to screen the candidate extreme point set to obtain a set of target extreme points, wherein each target extreme point in the target extreme point set corresponds to target point information; a first alignment module, which is used to align each target image in the target image data based on the target extreme point set using the central processing unit and the graphics processing unit to obtain aligned image data.

[0020] Furthermore, the second processing module includes: a first processing submodule, which is used to perform Gaussian filtering on the target image data on the graphics processor to obtain a pyramid image, and transmit the pyramid image to the central processing unit, wherein the pyramid image includes: multiple layers of images, and the resolution scale of each layer of the image is different; a second processing submodule, which is used to perform differential processing on the pyramid image on the central processing unit to obtain a differential pyramid image, wherein the differential pyramid image includes: multiple layers of differential images, and each layer of the differential image is an image obtained by differentially performing two adjacent layers of images in the pyramid image; a first determination submodule, which is used to determine, for each layer of the differential image, on the central processing unit based on the upper and lower layers of differential images adjacent to the differential image, candidate extreme value points on the differential image; a second determination submodule, which is used to determine the position and resolution scale of each candidate extreme value point, obtain point information of each candidate extreme value point, and add all candidate extreme value points to the candidate extreme value point set.

[0021] Furthermore, the first screening module includes: a first updating submodule, used to transmit the set of candidate extreme points to the graphics processor, and update the position of each candidate extreme point on the graphics processor based on the point information corresponding to each candidate extreme point; a third determining submodule, used to determine the second-order derivative matrix of each candidate extreme point on the graphics processor, and determine the principal curvature of each candidate extreme point based on the second-order derivative matrix; the first screening submodule, used to screen the target extreme points indicated by the principal curvature less than the preset principal curvature threshold from the extreme point set on the graphics processor to obtain the target extreme point set.

[0022] Furthermore, the first alignment module includes: a fourth determination submodule, used to transmit the target extreme point set to the central processing unit, and determine the target direction of each target extreme point on the central processing unit based on the target point information corresponding to each target extreme point; a fifth determination submodule, used to determine the feature descriptor of each target extreme point on the central processing unit based on the target direction of each target extreme point; and a first alignment submodule, used to transmit the feature descriptor of each target extreme point to the graphics processor, and align each target image in the target image data on the graphics processor based on the feature descriptor of each target extreme point to obtain aligned image data.

[0023] Furthermore, the first alignment submodule includes: a sixth determination submodule, used to determine the similarity value between every two target extreme value points based on the feature descriptor; a seventh determination submodule, used to determine, for each target extreme value point, another target extreme value point whose similarity value with the target extreme value point is the largest, and determine the target extreme value point and the other target extreme value point as a matching point pair; a second alignment submodule, used to align each target image in the target image data based on all matching point pairs to obtain aligned image data.

[0024] Furthermore, the detection device also includes: a first adding unit, which is used to add an attention structure and a pixel enhancement structure to the initial network structure before inputting the aligned image data into the preset land feature change detection model and outputting the detection result, so as to obtain a preset network structure, wherein the preset network structure also includes: an encoder and a decoder; the attention structure is used to perform weighted processing on each input aligned image to obtain a weighted feature map; the pixel enhancement structure is used to perform pixel enhancement processing on each input weighted feature map; a first acquisition unit, which is used to acquire historical image data, wherein the historical image data includes: a set of historical area images taken at multiple time points in different areas; a first labeling unit, which is used to label all historical area images in each historical area image set to obtain historical image labeling data; a first training unit, which is used to freeze the weight parameters of the encoder in the preset network structure, and use the historical image labeling data to iteratively train the weight parameters of the decoder in the preset network structure to obtain a preset land feature change detection model.

[0025] Furthermore, the detection device also includes: a first weighting unit, which is used to perform weighted processing on each input aligned image through an attention structure to obtain a weighted feature map, and the first weighting unit includes: a first extraction module, which is used to extract features from each aligned image to obtain an original feature map of each aligned image; a first determination module, which is used to determine the weight of each feature point on the original feature map to obtain a weighted feature map; a first obtaining module, which is used to obtain an attention weight feature map based on the attention weight matrix and the weight feature map; and a first fusion module, which is used to fuse the attention weight feature map with the original feature map to obtain a weighted feature map.

[0026] Furthermore, the detection device also includes: a first enhancement unit, which is used to perform pixel enhancement processing on each input weighted feature map through a pixel enhancement structure, and the first enhancement unit includes: a first decomposition module, which is used to decompose each weighted feature map to obtain a superpixel set, wherein the superpixel set includes: multiple superpixels, each superpixel is a region composed of a group of pixel points with adjacent positions and similar features; a second extraction module, which is used to extract features from each superpixel to obtain pixel feature information of each superpixel; a first comparison module, which is used to compare each pixel feature information of adjacent weighted feature maps in chronological order to obtain a change block; a first merging module, which is used to merge all change blocks to obtain a change area, wherein the change area is used to generate a change image.

[0027] According to another aspect of an embodiment of the present invention, a computer program product is also provided, including a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, any one of the above-mentioned methods for detecting changes in land objects is implemented.

[0028] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement any one of the above-mentioned methods for detecting changes in ground objects.

[0029] In the present invention, image data of the target area is collected, and a central processing unit and a graphics processing unit are used to perform parallel registration processing on the image data to obtain aligned image data, which is input into a preset ground object change detection model and the detection result is output, thereby solving the technical problem of low efficiency in image registration in related technologies, resulting in low timeliness in detecting ground object changes.

[0030] In the present invention, by allocating multiple tasks included in the image registration process to the central processing unit and the graphics processing unit for processing, the efficiency of image registration can be improved and the registration accuracy can be ensured. Then, by adding an attention structure and a pixel enhancement structure to the preset ground object change detection model, the model accuracy can be improved, thereby achieving the technical effect of real-time and accurate detection of ground object changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0032] Figure 1 is a flow chart of an optional method for detecting ground feature changes according to an embodiment of the present invention;

[0033] Figure 2 is a schematic diagram of an optional regional change detection result for a remote sensing image according to an embodiment of the present invention;

[0034] Figure 3 is a schematic diagram of an optional SI FT feature descriptor construction process according to an embodiment of the present invention;

[0035] Figure 4 is a schematic diagram of an optional SI FT parallel algorithm according to an embodiment of the present invention;

[0036] Figure 5 is a schematic diagram of an optional device for detecting ground feature changes according to an embodiment of the present invention;

[0037] Figure 6 The hardware structure block diagram of an electronic device (or mobile device) for detecting a ground feature change according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0039] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0040] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) collected and involved in the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with the relevant laws, regulations and standards of the relevant regions, necessary confidentiality measures are taken, and public order and good customs are not violated, and corresponding operation entrances are provided for users to choose to authorize or refuse. For example, an interface is set between the system and the relevant users or organizations. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0041] Aiming at the problems of difficulty in detecting edge areas of objects and poor computational timeliness in large-scale remote sensing images, the present invention proposes a method for detecting object changes in multi-temporal remote sensing images, thereby solving the problems of low object change recognition accuracy and slow speed in current large-scale remote sensing image object change detection.

[0042] In the present invention, in terms of image registration, SI FT (Scale-Invariant Feature Transform, an algorithm for image feature detection and description) parallel computing technology based on CUDA (Compute Unified Device Architecture, a general parallel computing architecture) is adopted to ensure the timeliness of the entire change detection process without losing the registration accuracy. In addition, the change detection algorithm of the present invention adopts the attention mechanism, residual connection technology and scale-interconnected regional change detection algorithm, and utilizes the full-scale jump connection and deep supervision technology to improve the accuracy of ground object change detection. In order to solve the problem of low recall rate of small area change detection, the present invention adopts pixel enhancement technology, and improves the small area change detection capability of remote sensing images through superpixel sampling network, end-to-end pixel change network module, superpixelization and adaptive superpixel merging modules. In addition, in order to solve the problem of network training speed, the training acceleration technology based on pre-trained encoder is adopted to realize the rapid training of ground object change detection network.

[0043] The present invention is described in detail below in conjunction with various embodiments.

[0044] Embodiment 1

[0045] According to an embodiment of the present invention, an embodiment of a method for detecting changes in ground objects is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0046] Figure 1 is a flow chart of an optional method for detecting ground feature changes according to an embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0047] Step S101 : collecting image data of a target area, wherein the image data includes: a plurality of images of the target area taken at different time points.

[0048] In an embodiment of the present invention, in order to detect changes in land objects in a target area (such as a forest) (for example, to detect whether the forest has been cut down in recent years, etc.), multiple remote sensing images of the target area taken at different time points (such as different years and months) can be collected to obtain image data.

[0049] Step S102 , using a central processing unit and a graphics processing unit to perform parallel registration processing on the image data to obtain aligned image data, wherein the aligned image data includes: a plurality of aligned images aligned in space coordinates.

[0050] In an embodiment of the present invention, according to the characteristics of data processing by a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit) (for example, a CPU is good at processing data with dependencies, while a GPU is good at processing data with good independence), the task of performing parallel registration processing on image data can be divided into multiple small tasks, so that the central processing unit (CPU) and the graphics processing unit (GPU) can work together to perform parallel registration processing on multiple image data of a target area taken at different time points, thereby obtaining aligned image data, which includes multiple aligned images aligned in spatial coordinates.

[0051] In the embodiment of the present invention, image registration is a key step in change detection, which is used to ensure that images of the same area taken at different time points are aligned in spatial coordinates, so as to accurately compare and analyze changes in objects in the images.

[0052] For example, the CUDA-based SI FT parallel computing technology can be used to efficiently perform registration processing on the image data.

[0053] In this embodiment, image registration ensures the accurate alignment of images taken at different time points in spatial coordinates, providing the necessary conditions for subsequent change detection. Through parallel processing, the time required for registration can be reduced to meet the needs of real-time or fast processing.

[0054] Step S103, input the aligned image data into a preset ground object change detection model, and output the detection results, wherein the preset ground object change detection model at least includes: an attention structure and a pixel enhancement structure, and the detection results at least include: change images sorted in chronological order, and each change image is marked with a change sub-region.

[0055] In an embodiment of the present invention, the precisely registered aligned image data is input into a pre-built ground object change detection model to perform ground object change detection and annotation, and the detection results can be obtained quickly and accurately. The detection results include: change images sorted in chronological order, and each change image is annotated with a change sub-area.

[0056] In an embodiment of the present invention, the preset ground object change detection model includes at least an attention structure and a pixel enhancement structure, which can accurately identify and mark the changed areas of the ground objects in the image, even if these changes occur in small-scale areas.

[0057] In an embodiment of the present invention, the attention structure is used to introduce an attention mechanism into the ground object change detection model, which can dynamically adjust the model's attention to different areas of the input image, automatically identify and focus on key areas in the image, suppress background noise and false changes, and improve the accuracy of changing area identification.

[0058] In an embodiment of the present invention, the pixel enhancement structure enhances the model's ability to detect small area changes and improves the recall rate of change area detection through a superpixel sampling network, an end-to-end pixel change network module, superpixelization and adaptive superpixel merging.

[0059] In this embodiment, by using the preset ground object change detection model, the changed areas of the ground objects in the image can be automatically identified and annotated, even if these changed areas are small or interfered by complex backgrounds. In addition, by adding an attention structure and a pixel enhancement structure to the preset ground object change detection model, the robustness and accuracy of the model for complex scene change detection are improved.

[0060] Figure 2 is a schematic diagram of an optional regional change detection result for a remote sensing image according to an embodiment of the present invention, such as Figure 2 As shown in the figure, remote sensing images of the same area taken at different time points are shown (such as Figure 2 (a), (b), (c), (d), (e), (f), (g), (h), (i), and (j) in the figure respectively represent remote sensing images of the same area taken at different time points and sorted in chronological order) for ground object change detection. Figure 2 In (a) and (b), we can intuitively see the changes in the features (such as the changes in the shape of new buildings, roads or other features). Figure 2 In (c), (d), (e), (f), (g), (h), (i), and (j), rectangular boxes are used to mark the areas where the features in the area have changed at different time points.

[0061] In summary, by allocating multiple tasks included in the image registration process to the central processing unit and the graphics processing unit for processing, the efficiency of image registration can be improved and the registration accuracy can be ensured. Then, by adding an attention structure and a pixel enhancement structure to the preset ground object change detection model, the model accuracy can be improved, thereby achieving the technical effect of detecting ground object changes in real time and accurately, thereby solving the technical problem of low efficiency in image registration in related technologies, resulting in low timeliness in detecting ground object changes.

[0062] In order to improve the accuracy of determining the alignment of image data, in the method for detecting changes in land objects provided in Example 1 of the present application, the image data is stored in a central processing unit, and the image data is preprocessed on the central processing unit to obtain target image data, and the target image data is transmitted to a graphics processing unit; the target image data is processed by the central processing unit and the graphics processing unit to obtain a set of candidate extreme points, wherein each candidate extreme point in the set of candidate extreme points corresponds to point information, and the point information includes at least: the location of the candidate extreme point and the resolution scale corresponding to the candidate extreme point; the candidate extreme point set is screened by the central processing unit and the graphics processing unit to obtain a set of target extreme points, wherein each target extreme point in the set of target extreme points corresponds to target point information; based on the set of target extreme points, the central processing unit and the graphics processing unit are used to align each target image in the target image data to obtain aligned image data.

[0063] In the embodiment of the present invention, since the GPU is good at calculating high-density data, if the data with poor independence is calculated on the GPU, the algorithm execution delay and the like will occur. Therefore, it is necessary to divide the parallel registration processing task into multiple subtasks according to the GPU hardware characteristics and the calculation characteristics of the SI FT algorithm, so as to improve the efficiency of the registration processing through the collaborative processing of the GPU and the CPU. For example, the feature detection and feature point matching with no correlation are divided into the GPU for calculation, and the detection of SI FT feature points can be divided into multiple links, and the kernel functions (i.e., kernel functions) of the corresponding links are designed and implemented respectively.

[0064] In this embodiment, in each link of feature detection, the initialization work of image data with small amount of calculation can be done on the CPU side. Other links, such as the construction of Gaussian pyramid, the detection of extreme points, etc., require a large number of floating-point calculations, which are implemented by GPU. Since there is data connection between each link, the CPU side and the GPU side need to perform data transmission and communication. For the matching of SIFT feature points, since the creation of KD-Tree (K-Dimensional Tree, i.e., a tree-shaped data structure for data storage and search in multidimensional space) requires a large amount of logical condition judgment, it is placed on the CPU side, and the query and comparison of feature points on KD-Tree are placed on the GPU. In this way, the high-speed floating-point calculation and density calculation characteristics of GPU can be fully utilized to greatly improve processing efficiency.

[0065] In an embodiment of the present invention, the acquired image data may be first stored in a central processing unit (CPU). Then, the image data is preprocessed on the CPU to optimize the subsequent calculation process, reduce the calculation load of the GPU, and improve the processing efficiency. The preprocessing includes but is not limited to: data cleaning, data format conversion, etc., to ensure that the data meets the requirements of GPU parallel processing. After the preprocessing is completed, the obtained target image data is uploaded to a graphics processing unit (GPU) to utilize the parallel computing capability of the GPU for subsequent data processing.

[0066] In an embodiment of the present invention, after the target image data is transmitted to the GPU, the parallel computing capability of the GPU and the logical processing capability of the CPU are used to perform feature point detection on the image data, including Gaussian pyramid construction, detection of local extreme points, feature point positioning, etc., to obtain a set of candidate extreme points, each candidate extreme point in the set of candidate extreme points corresponds to point information, and the point information includes at least: the location of the candidate extreme point and the resolution scale corresponding to the candidate extreme point.

[0067] In an embodiment of the present invention, after obtaining the candidate extreme point set, further screening is required to remove those repeated or unimportant feature points to obtain a target extreme point set (each target extreme point in the target extreme point set corresponds to target point information).

[0068] In the embodiment of the present invention, the target extreme point set is used to accurately align the target image data. The alignment process is achieved by matching SI FT feature points, and the parallel computing characteristics of the GPU and the logical processing capabilities of the CPU are used to complete efficient matching of feature points, ensuring that images of the same area taken at different time points are accurately aligned in spatial coordinates, thereby obtaining aligned image data.

[0069] Here, the aligned image data refers to image data in which images of the same area taken at different time points are aligned in spatial coordinates after registration processing, providing a basis for subsequent change detection.

[0070] In this embodiment, through the collaborative work of the CPU and GPU, the image data is preprocessed, feature point detected, screened and registered, and finally the aligned image data is obtained, which can make full use of the logical processing power of the CPU and the parallel computing power of the GPU, significantly improve the speed and accuracy of image registration, and provide a high-quality data foundation for subsequent ground object change detection. Through parallel processing and efficient feature point screening, the real-time and accuracy requirements of change detection in large-scale excessive image processing can be met, and the efficiency and reliability of ground object change detection can be improved.

[0071] In order to improve the accuracy of determining the set of candidate extreme value points, in the method for detecting changes in land objects provided in the first embodiment of the present application, Gaussian filtering is performed on the target image data on the graphics processor to obtain a pyramid image, and the pyramid image is transmitted to the central processing unit, wherein the pyramid image includes: multiple layers of images, each layer of which has a different resolution scale; differential processing is performed on the pyramid image on the central processing unit to obtain a differential pyramid image, wherein the differential pyramid image includes: multiple layers of differential images, each layer of differential images is an image obtained by differentially performing two adjacent layers of images in the pyramid image; for each layer of differential images, the central processing unit determines candidate extreme value points on the differential image based on the upper and lower layer differential images adjacent to the differential image; the position and resolution scale of each candidate extreme value point are determined, the point information of each candidate extreme value point is obtained, and all candidate extreme value points are added to the set of candidate extreme value points.

[0072] In an embodiment of the present invention, the target image data may be first subjected to Gaussian filtering on a graphics processing unit (GPU) to construct a pyramid image. The pyramid image is composed of multiple layers of images, each layer affecting a different resolution scale, forming a multi-scale representation from coarse to fine, which provides a multi-scale perspective for subsequent local extreme point detection.

[0073] In an embodiment of the present invention, the process of constructing a pyramid image is as follows: a Gaussian filter may be used to blur an image (e.g., a new pixel value is calculated based on the values ​​around the pixel, so that the image can be smoothed) to obtain images of different resolution scales, and then the images of different resolution scales are constructed according to the resolution size (e.g., from low to high) to obtain a pyramid image.

[0074] In the embodiment of the present invention, the pyramid image is transmitted to the central processing unit, so that the pyramid image is differentially processed on the CPU, that is, two adjacent layers of images in the pyramid image are differentially processed to obtain a differential pyramid image. Each layer of differential image is obtained based on the differential of two adjacent layers of images in the pyramid image, so that the local extreme points in the image can be highlighted, providing a basis for subsequent feature point detection.

[0075] For example, after constructing the Gaussian pyramid, two adjacent scale images are subtracted. For example, there are two images of scale 1 and scale 2, and the resolution and details of scale 2 are less than those of scale 1. The image of scale 2 is enlarged (upsampled) to the same size as scale 1, and then the enlarged image of scale 2 is subtracted from the image of scale 1. The above subtraction process is repeated. For each scale image, a difference image (i.e., differential image) is obtained. These difference images are arranged from low to high according to scale, and a differential pyramid image is obtained. Each layer indicates which parts of the image have significant changes at that scale.

[0076] In the embodiment of the present invention, the CPU determines the candidate extreme value points on each layer of the differential image based on each layer of the differential pyramid image and its adjacent upper and lower layer differential images, so as to determine the candidate extreme value points on the differential image.

[0077] Exemplarily, for each pixel on the differential image, all pixels in the neighborhood of the pixel (i.e., the surrounding area centered on the pixel) and all pixels in the neighborhood of the pixel at the corresponding position in the previous and next layers of the layer where the pixel is located can be determined first. In this way, each pixel actually compares 26 neighbors (i.e., 8 neighbors in the current layer, 9 neighbors in the previous layer, and 9 neighbors in the next layer) to determine whether the pixel is a local extreme point. If a pixel is the maximum or minimum value among all its neighbors, it may be a local extreme point. In addition, in order to ensure that these points are prominent enough, a threshold needs to be set. If the value of the point is greater than this threshold, it is identified as a true local extreme point (i.e., a candidate extreme point).

[0078] In the embodiment of the present invention, the CPU can determine the specific location information (i.e., the coordinate information on the image) of each candidate extreme point and its resolution scale in the pyramid image, which constitutes the point information of the candidate extreme point. All these candidate extreme points and the corresponding point information will be collected and added to the candidate extreme point set for subsequent feature point screening and registration processing.

[0079] In this embodiment, through the collaborative work of the GPU and the CPU, multi-level and multi-scale preprocessing and feature point detection of the target image data are realized. First, the parallel computing power of the GPU is used to perform Gaussian filtering and construct a pyramid image to provide a multi-scale perspective for feature point detection. Subsequently, the CPU performs differential processing on the pyramid image to obtain a differential pyramid image to highlight local changes in the image and facilitate the detection of feature points. Finally, the CPU determines the candidate extreme points on each layer of the differential image, records their position and resolution scale information, and constructs a set of candidate extreme points, which provides a key basis for subsequent feature point screening, precise positioning and registration.

[0080] In order to improve the accuracy of determining the target extreme point set, in the method for detecting changes in land features provided in the first embodiment of the present application, the candidate extreme point set is transmitted to a graphics processor, and the position of each candidate extreme point is updated on the graphics processor based on the point information corresponding to each candidate extreme point; the second-order derivative matrix of each candidate extreme point is determined on the graphics processor, and the principal curvature of each candidate extreme point is determined based on the second-order derivative matrix; and the target extreme point indicated by the principal curvature less than a preset principal curvature threshold is screened from the extreme point set on the graphics processor to obtain the target extreme point set.

[0081] In the embodiment of the present invention, after the CPU completes the preliminary construction of the candidate extreme point set, the set is transmitted to the graphics processing unit (GPU). The GPU can efficiently process a large number of data points with its parallel processing capability and is an ideal platform for accurate positioning of feature points and calculation of principal curvature.

[0082] In an embodiment of the present invention, after receiving the set of candidate extreme points, the GPU can accurately perform secondary positioning on the coordinate position of each candidate extreme point based on the position and scale information of each extreme point in the differential pyramid image. For example, secondary fitting can be performed on the position coordinates to find the exact position of the extreme point on the image plane, which helps to improve the positioning accuracy of the feature points and ensure that they can stably describe the structure in the image.

[0083] In an embodiment of the present invention, the GPU can calculate the second-order derivative matrix of each candidate extreme point, and based on the second-order derivative matrix, the principal curvature of each candidate extreme point can be determined. The calculation of the principal curvature helps to distinguish true feature points from potential pseudo feature points, such as local extreme points caused by noise or illumination changes. For example, a 2×2 Hessian matrix (i.e., Hessian matrix or Hessian matrix) can be used to describe the rate of change of intensity around the extreme point. The Hessian matrix is ​​a second-order derivative matrix that contains the rate of change of intensity along the x, y directions, and the mixed direction of x and y, and then the eigenvalues ​​and eigenvectors of the Hessian matrix are calculated. The eigenvalue gives the curvature intensity, while the eigenvector indicates the curvature direction. The larger of the two eigenvalues ​​represents the principal curvature, which is the direction in which the intensity changes most significantly.

[0084] In an embodiment of the present invention, by analyzing the principal curvature, it can be determined whether the extreme point is in a stable area in the image. If the principal curvature is too high, it indicates that the extreme point is located in the noise area or edge of the image. Such extreme points may be unstable and easily affected by image transformation. Therefore, a threshold (i.e., a preset principal curvature threshold) can be set to retain only those extreme points whose principal curvature is lower than this threshold, thereby ensuring that these extreme points are still identifiable and matchable under different image transformations, that is, the GPU can filter out target extreme points from the set of candidate extreme points according to the preset principal curvature threshold, thereby obtaining a set of target extreme points. These target extreme points have stable principal curvatures and are an important basis for subsequent feature description and matching.

[0085] In this embodiment, the parallel computing capability of the GPU is used to achieve accurate positioning and principal curvature calculation of the initially detected candidate extreme points, thereby completing further screening of feature points. First, the GPU receives a set of candidate extreme points from the CPU, and updates the position based on the point information, thereby improving the accuracy of feature point detection. Then, the second-order derivative matrix of each candidate extreme point is calculated to determine its principal curvature, which is an important basis for judging the stability of the feature points. Finally, based on the preset principal curvature threshold, a stable set of target extreme points is screened out, providing more accurate and reliable feature point information for subsequent feature description and matching.

[0086] In order to further improve the accuracy of determining the aligned image data, in the method for detecting changes in ground objects provided in Example 1 of the present application, a set of target extreme points is transmitted to a central processing unit, and the target direction of each target extreme point is determined on the central processing unit based on the target point information corresponding to each target extreme point; the feature descriptor of each target extreme point is determined on the central processing unit based on the target direction of each target extreme point; the feature descriptor of each target extreme point is transmitted to a graphics processor, and each target image in the target image data is aligned on the graphics processor based on the feature descriptor of each target extreme point to obtain aligned image data.

[0087] In the embodiment of the present invention, after the GPU has completed the precise positioning and screening of the candidate extreme points and generated the target extreme point set, the set will be transmitted to the central processing unit (CPU). The CPU, with its logic control and data processing capabilities, will be responsible for determining the target direction (i.e., the main direction) of the target extreme point and calculating the feature descriptor.

[0088] In the embodiment of the present invention, on the CPU, the direction of each target extreme point is determined based on the target point information in the target extreme point set. The target direction is determined to construct a description of the environment around the extreme point to enhance the robustness and discrimination of the feature descriptor.

[0089] In an embodiment of the present invention, the gradients of all pixels in the neighborhood of the target extreme point, that is, the direction and rate of change of the image intensity, can be calculated, and then the gradients of each pixel are Gaussian weighted and accumulated to establish a directional histogram, and the peak value of the directional histogram is detected to determine the main direction of the target extreme point.

[0090] In the embodiment of the present invention, based on the target direction of each target extreme point, the CPU continues to calculate the feature descriptor. The feature descriptor is a multidimensional vector describing the local environment characteristics of the feature point, which contains information describing the stability and uniqueness of the feature point.

[0091] Exemplarily, the area near the target extreme point can be divided into 16 smaller sub-regions (i.e., 4x4 grids). Each sub-region is equivalent to a small "observation point" for collecting more detailed image information. In each sub-region, the gradient of the pixel is calculated again, and then the gradient direction histogram in each sub-region is counted, and then the gradient histogram information of each sub-region is combined into a descriptor. Usually, 8 directional information of each histogram can be extracted, so each sub-region produces an 8-dimensional vector. By splicing 16 such vectors together, a 128-dimensional feature descriptor is obtained. This descriptor is like a "fingerprint" of the feature point, which can uniquely describe the image structure around the feature point. Finally, the feature descriptor can be normalized to ensure that it is not affected by changes in image brightness or contrast, making the feature descriptor clearer and more stable, which is convenient for subsequent feature matching.

[0092] In the embodiment of the present invention, since the calculation intensity of the 128-dimensional SI FT feature descriptor of the target extreme point is not large, more control flow instructions are used and there are more branch paths, which seriously affect the effective instruction throughput and greatly affect the execution efficiency of the GPU algorithm. Therefore, the construction of the SI FT descriptor is implemented on the CPU, which can give play to the advantages of the CPU's strong logical branching and judgment capabilities and complete the computing tasks more efficiently.

[0093] Figure 3 is a schematic diagram of an optional SI FT feature descriptor construction process according to an embodiment of the present invention, such as Figure 3 As shown in the figure, Block is the basic unit for CPU to execute tasks. Each Block can contain multiple Threads to process various parts of the task in parallel. Each Thread is responsible for processing a feature descriptor (i.e., one Thread corresponds to one feature vector). In the construction of SI FT feature descriptor, Thread is responsible for calculating the local environment information of the extreme point, such as gradient direction and magnitude, and finally forms a 128-dimensional feature vector, i.e., feature descriptor.

[0094] In an embodiment of the present invention, the feature descriptor of each target extreme point is transmitted from the CPU back to the GPU, and the GPU uses these feature descriptors to perform feature matching on the target image data to achieve image alignment. Feature matching can accelerate image registration and improve the accuracy and efficiency of change detection by comparing the feature descriptors of the target extreme points in different images and finding the best matching point pair. Finally, the GPU aligns the images at different time points by geometrically transforming the matching points to generate aligned image data.

[0095] In order to improve the accuracy of aligning each target image in the target image data, in the method for detecting changes in ground objects provided in Example 1 of the present application, based on the feature descriptor, the similarity value between every two target extreme points is determined; for each target extreme point, another target extreme point whose similarity value with the target extreme point is the largest is determined, and the target extreme point and the other target extreme point are determined as a matching point pair; based on all matching point pairs, each target image in the target image data is aligned to obtain aligned image data.

[0096] In the embodiment of the present invention, after the calculation of the feature descriptors is completed, the similarity between each two target extreme value points can be determined based on the feature descriptors. The similarity calculation is completed by comparing the distance or similarity between the two feature descriptors, for example, methods such as Euclidean distance, Manhattan distance or cosine similarity can be used.

[0097] After determining the similarity values ​​between all target extreme points, for each target extreme point, another target extreme point with the greatest similarity will be found, and these two extreme points will be determined as a matching point pair to ensure that each extreme point is paired with its best matching extreme point in the other image, thereby achieving accurate feature point matching.

[0098] After all matching point pairs are determined, the target image data will be aligned based on these matching point pairs. For example, the matching points are used to perform geometric transformations on the images, such as rotation and translation, so that images at different time points are accurately aligned in space, ensuring the spatial consistency between two or more images, so that subsequent ground object change detection can be performed accurately.

[0099] Figure 4 is a schematic diagram of an optional SI FT parallel algorithm according to an embodiment of the present invention, such as Figure 4As shown, Gaussian kernel functions of different scales can be transferred from the memory of the CPU host side to the constant memory of the GPU device side. Then the CPU host side initializes the image data and transfers the initialized image data to the GPU device side. The GPU device side can use the corresponding Gaussian kernel function to perform Gaussian filtering on the image data to establish a Gaussian pyramid image, and store the Gaussian pyramid image in the global memory to achieve data reuse and facilitate parallel computing using the GPU. The Gaussian pyramid image is downloaded back to the memory of the CPU host side. Based on the Gaussian pyramid image, the CPU host side subtracts Gaussian pyramids of adjacent scales to establish a Gaussian difference pyramid image. Then, based on the Gaussian difference pyramid image, the CPU host side locates the candidate feature points and records the features (such as the location) and the corresponding x and y scales of the candidate feature points. The calculated candidate feature point information is uploaded to the GPU device side to accurately locate the feature points in the GPU. Then, using the feature point information calculated in the GPU, another Gaussian weight function is started to calculate the gradient size and direction of the pixels near the feature point (i.e., feature M (gradient size), θ (feature direction)). The gradient size of each pixel in the neighborhood window of the feature point is Gaussian weighted and accumulated to establish a direction histogram, detect the peak of the direction histogram, and extract the main direction of the feature point. After that, the CPU host extracts the SI FT feature vector according to the main direction of the feature point to obtain the feature description operator, and then performs SI FT feature matching on the GPU device through the feature description operator.

[0100] In order to improve the accuracy of the detection results of ground object changes, it is necessary to construct a preset ground object change detection model. In the ground object change detection method provided in Example 1 of the present application, before the aligned image data is input into the preset ground object change detection model and the detection result is output, an attention structure and a pixel enhancement structure are added to the initial network structure to obtain a preset network structure, wherein the preset network structure also includes: an encoder and a decoder; the attention structure is used to perform weighted processing on each input aligned image to obtain a weighted feature map; the pixel enhancement structure is used to perform pixel enhancement processing on each input weighted feature map; historical image data is collected, wherein the historical image data includes: a set of historical area images taken at multiple time points in different areas; all historical area images in each historical area image set are labeled to obtain historical image annotation data; the weight parameters of the encoder in the preset network structure are frozen, and the weight parameters of the decoder in the preset network structure are iteratively trained using the historical image annotation data to obtain the preset ground object change detection model.

[0101] In an embodiment of the present invention, in order to address the problem that the light and shadow conditions of remote sensing images vary greatly and background noise is easily misdetected, which affects the accuracy of regional change detection, an attention structure and a pixel enhancement structure are introduced on the initial network structure (such as the U-Net structure, which is a convolutional neural network architecture) to construct a preset network structure.

[0102] In the embodiment of the present invention, the attention structure is mainly responsible for weighted processing of each input aligned image. By learning the importance of different regions, the network can focus on the key information of ground object changes in the image, suppress the influence of edge parts and background noise, and thus improve the pertinence and effectiveness of feature extraction. The pixel enhancement structure is used to perform pixel enhancement processing on each input weighted feature map, the purpose of which is to enhance the detection capability of small area changes, especially for the detection of ground object edges and small-scale ground object changes.

[0103] In the embodiment of the present invention, the preset network structure further includes an encoder and a decoder. The encoder is responsible for feature extraction, and the decoder is responsible for feature restoration and segmentation. The encoder gradually compresses the input image features through multiple downsampling operations while retaining multi-scale and multi-level information.

[0104] In the embodiment of the present invention, in order to train the ground object change detection model, a large amount of historical image data needs to be collected, which includes a set of historical regional images taken at different regions at multiple time points. The richness and diversity of historical image data are crucial to the generalization ability and robustness of the model, enabling the model to learn the patterns and characteristics of ground object changes in different environments.

[0105] In an embodiment of the present invention, the collected historical image data needs to be annotated, that is, the area of ​​land feature changes in each image is labeled to form historical image annotated data. The annotation process usually requires professional knowledge and personnel to ensure the accuracy and reliability of the annotation. Historical image annotated data is the key to training the model. It provides a supervisory signal to help the model learn how to correctly identify land feature changes.

[0106] In an embodiment of the present invention, in order to solve the problem of network training speed, a training acceleration technology based on a pre-trained encoder is adopted to realize the rapid training function of the ground object change detection network. Therefore, in the preset network structure, the weight parameters of the encoder can be frozen first, that is, the encoder part remains unchanged, and its pre-trained feature extraction capability will not be updated during the training process. Subsequently, the weight parameters of the decoder are iteratively trained using the collected historical image annotation data. The decoder is responsible for restoring the feature map output by the encoder to the change detection result. By training the decoder, the model can learn how to recover the details of the ground object changes from the feature map and generate reliable detection outputs.

[0107] In order to improve the accuracy of determining the weighted feature map, in the method for detecting changes in ground objects provided in Example 1 of the present application, feature extraction is performed on each aligned image to obtain the original feature map of each aligned image; the weight of each feature point on the original feature map is determined to obtain a weighted feature map; based on the attention weight matrix and the weighted feature map, an attention weighted feature map is obtained; and the attention weighted feature map is fused with the original feature map to obtain a weighted feature map.

[0108] In an embodiment of the present invention, after the aligned image data is input into a preset object change detection model, the attention structure can extract features from each image to identify and encode the objects and change information in the image, and obtain the original feature map of each aligned image. That is, the attention structure converts the aligned image into a multi-scale feature representation through a series of convolution, activation, and pooling layers. The feature map of each level contains information of different scales and levels, which is crucial for subsequent change detection.

[0109] In an embodiment of the present invention, after the original feature maps are generated, these feature maps are further processed to assign weights to each feature point. The weights are determined through an attention mechanism that learns the importance of each position in the image so that it can distinguish key information from background noise. The generation of weighted feature maps enables the model to focus on more important ground feature change information while suppressing the influence of irrelevant or noisy areas.

[0110] Then, the attention weight matrix (learned through the attention mechanism, used to represent the weight of each position) is combined with the weight feature map, and the attention weight feature map is obtained by matrix multiplication or dot multiplication. That is, by performing secondary weighting on the feature points, the features of the changed area are further highlighted, while the irrelevant areas are further suppressed.

[0111] After that, the attention weight feature map is fused with the original feature map to generate a weighted feature map. This fusion process can be achieved through simple weighted summation, dot product or other mathematical operations. Through fusion, the model can simultaneously utilize the information in the original feature map and the key change information emphasized by the attention mechanism, thereby generating a more comprehensive and accurate feature representation, providing richer and more robust feature data for subsequent change detection.

[0112] In order to improve the accuracy of pixel enhancement processing for each input weighted feature map, in the method for detecting changes in ground objects provided in Example 1 of the present application, each weighted feature map is decomposed to obtain a superpixel set, wherein the superpixel set includes: multiple superpixels, each superpixel is a region composed of a group of pixels with adjacent positions and similar features; feature extraction is performed on each superpixel to obtain pixel feature information of each superpixel; each pixel feature information of adjacent weighted feature maps is compared in chronological order to obtain a change block; all change blocks are merged to obtain a change region, wherein the change region is used to generate a change image.

[0113] In an embodiment of the present invention, each weighted feature map can be decomposed to segment the image into multiple superpixels to obtain a superpixel set. A superpixel is a clustering result of a pixel, and each superpixel is a region composed of a group of pixels with adjacent positions and similar features. Through superpixel segmentation, the amount of calculation in subsequent processing can be reduced while maintaining the coherence and integrity of the area where the object changes.

[0114] After decomposing the weighted feature map into a set of superpixels, feature extraction can be performed on each superpixel, so that more concentrated pixel feature information can be extracted for each superpixel, including attributes such as texture, texture, and color. Feature extraction helps reduce dimensions while maintaining features that are critical for change detection, allowing the model to identify changes in objects more efficiently and accurately.

[0115] After obtaining the pixel feature information of each superpixel, the adjacent weighted feature maps can be compared in chronological order. This comparison process can locate the location of the ground object change by comparing the feature information of the corresponding superpixels in the images of different phases. The comparison results will mark the superpixel areas in the image that have changed, and these areas are defined as change blocks. Then, all the detected change blocks are merged to form a change area. This merging process ensures the integrity and coherence of the ground object changes and avoids the isolation and fragmentation between the change blocks.

[0116] In the embodiment of the present invention, the change area is used to generate a change image, that is, an image that clearly marks the specific location and range of the change of the ground object.

[0117] In the embodiment of the present invention, in terms of image registration, the CUDA-based SI FT parallel computing technology is adopted, which can ensure the timeliness of the entire change detection process without losing the registration accuracy. In addition, the attention mechanism is introduced in the change detection algorithm, thereby improving the accuracy of ground object change detection. In addition, in order to solve the problem of low recall rate of small area change detection, pixel enhancement technology is introduced, and the ability to detect small area changes in remote sensing images is improved through modules such as superpixel sampling network, end-to-end pixel change network module, superpixelization and adaptive superpixel merging. In addition, in order to solve the problem of network training speed, a training acceleration technology based on a pre-trained encoder is adopted to achieve rapid training of the regional change detection network.

[0118] The following is a detailed description in conjunction with another embodiment.

[0119] Embodiment 2

[0120] A device for detecting changes in ground objects provided in this embodiment includes a plurality of implementation units, each of which corresponds to each implementation step in the above-mentioned first embodiment.

[0121] Figure 5 is a schematic diagram of an optional device for detecting ground feature changes according to an embodiment of the present invention, such as Figure 5 As shown, the detection device may include: a collection unit 50, a processing unit 51, and an output unit 52.

[0122] The acquisition unit 50 is used to acquire image data of the target area, wherein the image data includes: a plurality of images of the target area taken at different time points;

[0123] The processing unit 51 is used to perform parallel registration processing on the image data using a central processing unit and a graphics processing unit to obtain aligned image data, wherein the aligned image data includes: a plurality of aligned images aligned in space coordinates;

[0124] The output unit 52 is used to input the aligned image data into a preset ground object change detection model and output the detection results, wherein the preset ground object change detection model includes at least: an attention structure and a pixel enhancement structure, and the detection results include at least: change images sorted in chronological order, and each change image is marked with a change sub-region.

[0125] The above detection device can improve the efficiency of image registration and ensure the registration accuracy by allocating multiple tasks included in the image registration process to the central processing unit and the graphics processing unit for processing. Then, by adding an attention structure and a pixel enhancement structure to the preset ground object change detection model, the model accuracy can be improved, thereby achieving the technical effect of real-time and accurate detection of ground object changes, thereby solving the technical problem of low efficiency in image registration in related technologies, resulting in low timeliness in detecting ground object changes.

[0126] Furthermore, the processing unit includes: a first processing module, which is used to store the image data in a central processing unit, and pre-process the image data on the central processing unit to obtain target image data, and transmit the target image data to a graphics processing unit; a second processing module, which is used to use the central processing unit and the graphics processing unit to process the target image data to obtain a set of candidate extreme points, wherein each candidate extreme point in the candidate extreme point set corresponds to point information, and the point information at least includes: the location of the candidate extreme point and the resolution scale corresponding to the candidate extreme point; a first screening module, which is used to use the central processing unit and the graphics processing unit to screen the candidate extreme point set to obtain a set of target extreme points, wherein each target extreme point in the target extreme point set corresponds to target point information; a first alignment module, which is used to align each target image in the target image data based on the target extreme point set using the central processing unit and the graphics processing unit to obtain aligned image data.

[0127] Furthermore, the second processing module includes: a first processing submodule, which is used to perform Gaussian filtering on the target image data on the graphics processor to obtain a pyramid image, and transmit the pyramid image to the central processing unit, wherein the pyramid image includes: multiple layers of images, and the resolution scale of each layer of the image is different; a second processing submodule, which is used to perform differential processing on the pyramid image on the central processing unit to obtain a differential pyramid image, wherein the differential pyramid image includes: multiple layers of differential images, and each layer of the differential image is an image obtained by differentially performing two adjacent layers of images in the pyramid image; a first determination submodule, which is used to determine, for each layer of the differential image, on the central processing unit based on the upper and lower layers of differential images adjacent to the differential image, candidate extreme value points on the differential image; a second determination submodule, which is used to determine the position and resolution scale of each candidate extreme value point, obtain point information of each candidate extreme value point, and add all candidate extreme value points to the candidate extreme value point set.

[0128] Furthermore, the first screening module includes: a first updating submodule, used to transmit the set of candidate extreme points to the graphics processor, and update the position of each candidate extreme point on the graphics processor based on the point information corresponding to each candidate extreme point; a third determining submodule, used to determine the second-order derivative matrix of each candidate extreme point on the graphics processor, and determine the principal curvature of each candidate extreme point based on the second-order derivative matrix; the first screening submodule, used to screen the target extreme points indicated by the principal curvature less than the preset principal curvature threshold from the extreme point set on the graphics processor to obtain the target extreme point set.

[0129] Furthermore, the first alignment module includes: a fourth determination submodule, used to transmit the target extreme point set to the central processing unit, and determine the target direction of each target extreme point on the central processing unit based on the target point information corresponding to each target extreme point; a fifth determination submodule, used to determine the feature descriptor of each target extreme point on the central processing unit based on the target direction of each target extreme point; and a first alignment submodule, used to transmit the feature descriptor of each target extreme point to the graphics processor, and align each target image in the target image data on the graphics processor based on the feature descriptor of each target extreme point to obtain aligned image data.

[0130] Furthermore, the first alignment submodule includes: a sixth determination submodule, used to determine the similarity value between every two target extreme value points based on the feature descriptor; a seventh determination submodule, used to determine, for each target extreme value point, another target extreme value point whose similarity value with the target extreme value point is the largest, and determine the target extreme value point and the other target extreme value point as a matching point pair; a second alignment submodule, used to align each target image in the target image data based on all matching point pairs to obtain aligned image data.

[0131] Furthermore, the detection device also includes: a first adding unit, which is used to add an attention structure and a pixel enhancement structure to the initial network structure before inputting the aligned image data into the preset land feature change detection model and outputting the detection result, so as to obtain a preset network structure, wherein the preset network structure also includes: an encoder and a decoder; the attention structure is used to perform weighted processing on each input aligned image to obtain a weighted feature map; the pixel enhancement structure is used to perform pixel enhancement processing on each input weighted feature map; a first acquisition unit, which is used to acquire historical image data, wherein the historical image data includes: a set of historical area images taken at multiple time points in different areas; a first labeling unit, which is used to label all historical area images in each historical area image set to obtain historical image labeling data; a first training unit, which is used to freeze the weight parameters of the encoder in the preset network structure, and use the historical image labeling data to iteratively train the weight parameters of the decoder in the preset network structure to obtain a preset land feature change detection model.

[0132] Furthermore, the detection device also includes: a first weighting unit, which is used to perform weighted processing on each input aligned image through an attention structure to obtain a weighted feature map, and the first weighting unit includes: a first extraction module, which is used to extract features from each aligned image to obtain an original feature map of each aligned image; a first determination module, which is used to determine the weight of each feature point on the original feature map to obtain a weighted feature map; a first obtaining module, which is used to obtain an attention weight feature map based on the attention weight matrix and the weight feature map; and a first fusion module, which is used to fuse the attention weight feature map with the original feature map to obtain a weighted feature map.

[0133] Furthermore, the detection device also includes: a first enhancement unit, which is used to perform pixel enhancement processing on each input weighted feature map through a pixel enhancement structure, and the first enhancement unit includes: a first decomposition module, which is used to decompose each weighted feature map to obtain a superpixel set, wherein the superpixel set includes: multiple superpixels, each superpixel is a region composed of a group of pixel points with adjacent positions and similar features; a second extraction module, which is used to extract features from each superpixel to obtain pixel feature information of each superpixel; a first comparison module, which is used to compare each pixel feature information of adjacent weighted feature maps in chronological order to obtain a change block; a first merging module, which is used to merge all change blocks to obtain a change area, wherein the change area is used to generate a change image.

[0134] The above-mentioned detection device may also include a processor and a memory. The above-mentioned acquisition unit 50, processing unit 51, output unit 52, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0135] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the aligned image data is input into the preset ground object change detection model by adjusting the kernel parameters to output the detection results.

[0136] The above-mentioned memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory (fl ash RAM), and the memory includes at least one storage chip.

[0137] The present invention also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that initializes the following method steps: collecting image data of the target area, using a central processing unit and a graphics processing unit to perform parallel registration processing on the image data to obtain aligned image data, inputting the aligned image data into a preset ground object change detection model, and outputting the detection results.

[0138] According to another aspect of an embodiment of the present invention, a computer program product is also provided, including a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, any one of the above-mentioned methods for detecting changes in land objects is implemented.

[0139] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement the above-mentioned method for detecting changes in ground objects.

[0140] Figure 6 FIG. 1 is a hardware structure block diagram of an electronic device (or mobile device) for detecting a ground feature change according to an embodiment of the present invention. Figure 6 As shown, the electronic device may include one or more processors (e.g., Figure 6 The processor 602a, processor 602b, ..., processor 602n, etc., which may include but are not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 604 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a keyboard, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 6 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 6 More or fewer components as shown, or with Figure 6 Different configurations are shown.

[0141] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0142] The embodiments or examples of the present disclosure are not exhaustive, but are only illustrative of some embodiments or examples, and are not intended to be specific limitations on the scope of protection of the present disclosure. In the absence of contradiction, each step in a certain embodiment or example can be implemented as an independent example, and the steps can be combined arbitrarily. For example, the scheme after removing some steps in a certain embodiment or example can also be implemented as an independent example, and the order of the steps in a certain embodiment or example can be arbitrarily exchanged. In addition, the optional methods or optional examples in a certain embodiment or example can be combined arbitrarily; in addition, the various embodiments or examples can be combined arbitrarily, for example, some or all steps of different embodiments or examples can be combined arbitrarily, and a certain embodiment or example can be combined arbitrarily with the optional methods or optional examples of other embodiments or examples.

[0143] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0144] In the several embodiments provided by the present invention, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0145] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0146] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0147] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0148] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for detecting ground feature changes, characterized in that: include: Collecting image data of the target area, wherein the image data includes: a plurality of images of the target area taken at different time points; Using a central processing unit and a graphics processing unit, the image data is parallelly registered to obtain aligned image data, wherein the aligned image data includes: a plurality of aligned images aligned in space coordinates; The aligned image data is input into a preset ground object change detection model, and a detection result is output, wherein the preset ground object change detection model includes at least: an attention structure and a pixel enhancement structure, and the detection result includes at least: change images sorted in chronological order, and each of the change images is marked with a change sub-region.

2. The detection method according to claim 1, characterized in that: Using CPU and GPU, The step of performing parallel registration processing on the image data to obtain aligned image data comprises: The image data is stored in a central processing unit, and the image data is pre-processed on the central processing unit to obtain target image data, and the target image data is transmitted to a graphics processing unit; The target image data is processed by the central processing unit and the graphics processing unit to obtain a set of candidate extreme value points, wherein each candidate extreme value point in the set of candidate extreme value points corresponds to point information, and the point information at least includes: the position of the candidate extreme value point and the resolution scale corresponding to the candidate extreme value point; Using the central processing unit and the graphics processing unit, the candidate extreme value point set is screened to obtain a target extreme value point set, wherein each target extreme value point in the target extreme value point set corresponds to target point information; Based on the target extreme point set, the central processing unit and the graphics processing unit are used to align each target image in the target image data to obtain the aligned image data.

3. The detection method according to claim 2, characterized in that: The step of using the central processing unit and the graphics processing unit to process the target image data to obtain a set of candidate extreme value points includes: Performing Gaussian filtering on the target image data on the graphics processor to obtain a pyramid image, and transmitting the pyramid image to the central processor, wherein the pyramid image includes: multiple layers of images, each layer of images having a different resolution scale; Performing differential processing on the pyramid image on the central processing unit to obtain a differential pyramid image, wherein the differential pyramid image includes: multiple layers of differential images, each layer of the differential image is an image obtained by performing differential processing on two adjacent layers of images in the pyramid image; For each layer of the differential image, determining the candidate extreme value points on the differential image based on the upper and lower layer differential images adjacent to the differential image on the central processing unit; Determine the position and resolution scale of each of the candidate extreme value points, obtain the point information of each of the candidate extreme value points, and add all the candidate extreme value points to the candidate extreme value point set.

4. The detection method according to claim 2, characterized in that: The step of using the central processing unit and the graphics processing unit to screen the candidate extreme point set to obtain a target extreme point set includes: The set of candidate extreme value points is transmitted to the graphics processor, and the position of each candidate extreme value point is updated on the graphics processor based on the point information corresponding to each candidate extreme value point; Determining a second-order derivative matrix of each of the candidate extreme value points on the graphics processor, and determining a principal curvature of each of the candidate extreme value points based on the second-order derivative matrix; The target extreme value points indicated by the principal curvatures less than a preset principal curvature threshold are screened from the extreme value point set on the graphics processor to obtain the target extreme value point set.

5. The detection method according to claim 2, characterized in that: Based on the target extreme point set, the central processing unit and the graphics processing unit are used to align each target image in the target image data to obtain the aligned image data, comprising: The target extreme value point set is transmitted to the central processor, and the target direction of each target extreme value point is determined on the central processor based on the target point information corresponding to each target extreme value point; Determining, on the central processor, a feature descriptor of each of the target extreme value points based on the target direction of each of the target extreme value points; The feature descriptor of each target extreme point is transmitted to the graphics processor, and each target image in the target image data is aligned on the graphics processor based on the feature descriptor of each target extreme point to obtain aligned image data.

6. The detection method according to claim 5, characterized in that: The step of aligning each target image in the target image data based on the feature descriptor of each target extreme point on the graphics processor to obtain aligned image data comprises: Based on the feature descriptor, determining a similarity value between every two target extreme value points; For each of the target extreme value points, determine another target extreme value point whose similarity value with the target extreme value point is the largest, and determine the target extreme value point and the another target extreme value point as a matching point pair; Based on all the matching point pairs, each of the target images in the target image data is aligned to obtain the aligned image data.

7. The detection method according to claim 1, characterized in that: Before inputting the aligned image data into a preset ground object change detection model and outputting the detection result, the method further includes: The attention structure and the pixel enhancement structure are added to the initial network structure to obtain a preset network structure, wherein the preset network structure also includes: an encoder and a decoder; the attention structure performs weighted processing on each of the input aligned images to obtain a weighted feature map; the pixel enhancement structure performs pixel enhancement processing on each of the input weighted feature maps; Collecting historical image data, wherein the historical image data includes: a set of historical regional images taken at multiple time points in different regions; Annotating all the historical area images in each of the historical area image sets to obtain historical image annotation data; The weight parameters of the encoder in the preset network structure are frozen, and the weight parameters of the decoder in the preset network structure are iteratively trained using the historical image annotation data to obtain the preset ground object change detection model.

8. The detection method according to claim 7, characterized in that: The attention structure performs weighted processing on each of the input aligned images to obtain a weighted feature map, including: Performing feature extraction on each of the aligned images to obtain an original feature map of each of the aligned images; Determine the weight of each feature point on the original feature map to obtain a weighted feature map; Based on the attention weight matrix and the weight feature map, obtaining an attention weight feature map; The attention weight feature map is fused with the original feature map to obtain the weighted feature map.

9. The detection method according to claim 7, characterized in that: The pixel enhancement structure performs pixel enhancement processing on each of the input weighted feature maps, including: Decomposing each weighted feature map to obtain a superpixel set, wherein the superpixel set includes: a plurality of superpixels, each of which is a region composed of a group of pixel points that are adjacent in position and have similar features; Performing feature extraction on each of the superpixels to obtain pixel feature information of each of the superpixels; Compare the pixel feature information of each of the adjacent weighted feature maps in chronological order to obtain a change block; All the change blocks are merged to obtain a change region, wherein the change region is used to generate the change image.

10. A device for detecting changes in ground objects, characterized in that: include: A collection unit, used for collecting image data of a target area, wherein the image data includes: a plurality of images of the target area taken at different time points; A processing unit, configured to use a central processing unit and a graphics processing unit to perform parallel registration processing on the image data to obtain aligned image data, wherein the aligned image data includes: a plurality of aligned images aligned in space coordinates; An output unit is used to input the aligned image data into a preset ground object change detection model and output a detection result, wherein the preset ground object change detection model at least includes: an attention structure and a pixel enhancement structure, and the detection result at least includes: change images sorted in chronological order, and each of the change images is annotated with a change sub-region.

Images