Instance-constrained change detection method and device for broken image patches of remote sensing images

Abstract

The application provides an instance constraint change detection method and device for broken image patches of remote sensing images, and relates to the technical field of intelligent remote sensing image processing. The method comprises the following steps: performing image slicing on the obtained front and rear time-phase images and inputting the images into a pre-trained instance constraint change detection network (including a backbone network, a feature fusion module, an instance constraint decoding module and an instance constraint loss, the instance constraint loss is used to calculate the loss of each change patch in the image during model training, and the loss of all change patches is accumulated for gradient update), to obtain a predicted change map containing the patches; then, small patch filtering, hole filling and contour simplification are performed to obtain a patch optimization result; finally, raster filling and vector conversion operations are performed to generate a target change vector result. The application can accurately detect the change area, ensure the accuracy and recall rate, and take into account the integrity and smoothness of the change patch, thereby significantly improving the precision of change detection.

Description

Technical Field

[0001] This application relates to the field of intelligent processing technology for remote sensing images, and in particular to a method and apparatus for detecting instance constraint changes in broken patches of remote sensing images. Background Technology

[0002] Currently, with the development of remote sensing satellite imaging technology, the detection of Earth-related changes is playing an increasingly important role in fields such as urban planning, environmental monitoring, agricultural surveys, and disaster assessment.

[0003] In related technologies, traditional change detection methods that rely on deep learning are commonly used, or methods that utilize the feature learning capabilities of artificial intelligence neural networks to automatically model the relationship between image objects and geographic change features for change detection.

[0004] However, traditional change detection methods have limited ability to acquire spatial context information and complex visual features. On the other hand, relying on manual adjustment makes it difficult to set appropriate segmentation parameters, resulting in poor performance. The second type of change detection algorithm is designed for ground feature change tasks in complex scenes. Due to factors such as low image resolution, low registration accuracy, and differences in style between different time phases, it is difficult to effectively transfer and solve such complex remote sensing scene change detection problems. Especially for ground features with unclear boundaries, varying scales, and different shapes, this method cannot clearly and accurately describe their change boundaries, resulting in the obtained change patches being prone to missed detections and broken and irregular shapes. Summary of the Invention

[0005] The purpose of this application is to provide a method and apparatus for instance constraint change detection of fragmented patches in remote sensing images, which can accurately detect the changed areas, and while ensuring accuracy and recall, also take into account the integrity and smoothness of the changed patches, thus significantly improving the accuracy of change detection.

[0006] In a first aspect, the present invention provides a method for detecting instance constraint changes in fragmented patches of remote sensing images, comprising: acquiring a previous time-phase image and a subsequent time-phase image to be detected; slicing the previous time-phase image and the subsequent time-phase image; inputting the sliced ​​data into a pre-trained instance constraint change detection network to obtain a predicted change map containing the patches; wherein the instance constraint change detection network includes a backbone network, a feature fusion module, an instance constraint decoding module, and an instance constraint loss, the instance constraint loss being used to calculate the loss of each changed patch in the image during model training, and accumulating the losses of all changed patches for gradient update; performing small patch filtering and hole filling processing on the predicted change map, and simplifying the contours of the changed patches in the predicted change map to obtain patch optimization results; and performing raster filling and vector conversion operations on the patch optimization results to generate a target change vector result.

[0007] In an optional implementation, the backbone network is a swin-transformer structure used to extract image features at multiple scales; the sub-network structure used to extract features corresponding to each scale is composed of multiple stacked swin-blocks, and the number of swin-blocks corresponding to the sub-network structure at different scales is not exactly the same, and the swin-blocks adopt a moving window attention mechanism; the scale of the feature fusion module is set to correspond to the scale of the backbone network, and the feature fusion module is used to calculate the difference features by adding and averaging the feature maps of the previous and subsequent phases at different scales, and by subtracting them, and then concatenating the difference feature maps obtained by the two calculation methods in the channel dimension and using the convolution module to perform feature fusion.

[0008] In an optional implementation, the instance constraint decoding module includes: a pixel decoding module, a transformer decoder module, a multilayer perceptron module, and an instance loss; wherein, the pixel decoding module performs convolutional fusion on the difference feature maps of multiple scales output by the feature fusion module; the transformer decoder module performs mask feature query on the smallest highest-order feature map output by the feature fusion module, the transformer decoder module includes multiple network layers, each network layer includes a self-attention network, a cross-attention network, and a feedforward network, and the feature dimension of each network is preset with a corresponding target dimension; the multilayer perceptron module includes a category feature submodule and a mask feature submodule, the category feature submodule is used to generate the final predicted category score, and the mask feature submodule is used to generate the final mask feature map; the final category change map is generated by performing matrix multiplication on the predicted category score generated by the category feature submodule and the mask feature map generated by the mask feature submodule; the instance loss of the instance constraint decoding module includes classification loss and mask loss.

[0009] In an optional implementation, the instance constraint loss includes the change instance loss:

[0010]

[0011] Where N is the number of changed instances in the truth labels, w pos γ is the loss adjustment coefficient for varying instances, and γ is the loss scaling factor. For the i-th instance of the changed truth value, p pos For predicting the probability of change, ε is a parameter used to prevent loss calculation anomalies.

[0012] In an optional implementation, the instance constraint loss includes the background instance loss:

[0013]

[0014] Among them, wneg m is the background instance loss adjustment coefficient. gt_bg For background truth labels, p neg This is a probability map for the background.

[0015] In an optional implementation, image slicing is performed on the preceding and following temporal images, including: extracting image information corresponding to the preceding and following temporal images respectively, wherein the image information includes at least image coordinates, number of rows and columns, and resolution; calculating the image intersection region of the preceding and following temporal images based on the image information; and establishing slice index coordinates within the image intersection region according to the sliding window size and sliding window step size corresponding to the images, so as to perform image slicing based on the slice index coordinates.

[0016] In an optional implementation, the method further includes: calculating the resolution scaling factor of the previous and subsequent temporal images; if the resolution scaling factor is less than a preset threshold, performing preset optimization processing on the resolution scaling factor to obtain an optimized target scaling factor; and enlarging the sliding window size and sliding window step size of the higher resolution image based on the target scaling factor to reconstruct the slice index target based on the enlarged sliding window size and sliding window step size.

[0017] Secondly, the present invention provides an instance constraint change detection device for fragmented patches in remote sensing images, comprising: an image slicing module for acquiring previous and subsequent temporal images to be detected, and slicing the previous and subsequent temporal images; a network detection module for inputting the sliced ​​data into a pre-trained instance constraint change detection network to obtain a predicted change map containing the patches; wherein the instance constraint change detection network includes a backbone network, a feature fusion module, an instance constraint decoding module, and an instance constraint loss, the instance constraint loss being used to calculate the loss of each changed patch in the image during model training, and accumulating the losses of all changed patches for gradient update; a patch processing module for filtering small patches and filling holes in the predicted change map, and simplifying the contours of the changed patches in the predicted change map to obtain a patch optimization result; and a vector conversion module for raster filling and converting the patch optimization result into a vector to generate a target change vector result.

[0018] Thirdly, the present invention provides an electronic device including a processor and a memory, the memory storing computer-executable instructions that can be executed by the processor, the processor executing the computer-executable instructions to implement the instance constraint change detection method for broken patches in remote sensing images according to any of the foregoing embodiments.

[0019] Fourthly, the present invention provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are invoked and executed by a processor, the computer-executable instructions cause the processor to implement the instance constraint change detection method for broken patches in remote sensing images according to any of the foregoing embodiments.

[0020] The method and apparatus for instance-constrained change detection of fragmented patches in remote sensing images provided in this application acquire the previous and subsequent temporal images to be detected, and slice the previous and subsequent temporal images; input the sliced ​​data into a pre-trained instance-constrained change detection network to obtain a predicted change map containing the patches; wherein, the instance-constrained change detection network includes a backbone network, a feature fusion module, an instance-constrained decoding module, and an instance-constrained loss, which is used to calculate the loss of each changed patch in the image during model training, and the losses of all changed patches are accumulated for gradient update; small patch filtering and hole filling are performed on the predicted change map, and the contours of the changed patches in the predicted change map are simplified to obtain the patch optimization result; the patch optimization result is raster-filled and converted to vector to generate the target change vector result.

[0021] The above method uses a pre-trained instance-constrained change detection network to detect changes in the previous and subsequent time-phase images, which can improve the accuracy and detection effect of changed ground objects. By performing a series of processing on the patches (small patch filtering, hole filling and contour simplification), the completeness and smoothness of the detected changed patches can be improved, significantly improving the accuracy of change detection. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 A flowchart illustrating an instance constraint change detection method for fragmented patches in remote sensing images, provided as an embodiment of this application;

[0024] Figure 2 This is a schematic diagram of the structure of an instance constraint change detection network provided in an embodiment of this application;

[0025] Figure 3 A schematic diagram of a slice of remote sensing images at different resolutions for each phase of time, provided as an embodiment of this application;

[0026] Figure 4 This is a schematic diagram illustrating the small patch removal and cavity filling process provided in an embodiment of this application.

[0027] Figure 5 This application provides a schematic diagram of the optimization of the edge contour of a variable patch.

[0028] Figure 6 A flowchart outlining the execution of a method provided in this application embodiment;

[0029] Figure 7 A structural diagram of an example constraint change detection device for fragmented patches in remote sensing images provided in an embodiment of this application;

[0030] Figure 8 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0032] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0033] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0034] Change detection is an Earth observation task that observes the same spatial area at different times and identifies changes in targets within that area. Benefiting from the rapid development of remote sensing satellite imaging technology, the accumulation of massive amounts of remote sensing data, and the continuous development of artificial intelligence technology, change detection is playing an increasingly important role in fields such as urban planning, environmental monitoring, agricultural surveys, and disaster assessment.

[0035] Current change detection algorithms can be broadly categorized into two main types: traditional change detection methods and deep learning-based change detection methods. Traditional methods rely on manual feature design, threshold parameter adjustment, and feature modeling. For example, algebraic methods use image differencing, image regression, and Change Vector Analysis (CVA) to obtain change maps; classification-based methods use trained SVM classifiers or decision trees to directly classify pixels in the image and then compare the classification maps from two temporal phases to obtain the final change map. The second type is deep learning-based methods, which utilize the feature learning capabilities of artificial intelligence neural networks to automatically model the relationship between image objects and geographic change features, thereby detecting more change information. Examples include methods that extend early fusion and Siamese architectures to fully convolutional networks such as FC-EF and FC-Siam-Conc, ChangeFormer built entirely using transformers, and ChangeStar, a single-temporal semi-supervised method.

[0036] The first type of algorithm has limited ability to acquire spatial context information and complex visual features. Furthermore, its reliance on manual adjustment makes it difficult to set suitable segmentation parameters, thus limiting performance. While the second type of algorithm significantly improves the ability to detect changing targets and its level of automation, it struggles to effectively address complex remote sensing scene change detection tasks in complex scenarios. Factors such as low image resolution, inaccurate registration, and stylistic differences between different time periods make it difficult for existing deep learning-based change detection methods to effectively transfer to such complex remote sensing scene change detection problems. In particular, for features with unclear boundaries, varying scales, and diverse shapes, this method cannot clearly and accurately describe their change boundaries, leading to frequent missed detections and fragmented, irregular shapes in the resulting change patches. Ultimately, this affects the accuracy of land use change surveys and natural resource change monitoring.

[0037] Based on this, the embodiments of this application provide an instance constraint change detection method and apparatus for fragmented patches in remote sensing images, which addresses the problems of missed and false detections of changed ground features and unsatisfactory detection results. It can accurately detect changed areas, and while ensuring accuracy and recall, it also takes into account the integrity and smoothness of the changed patches, thus significantly improving the accuracy of change detection.

[0038] This application provides a method for detecting instance constraint changes in fragmented patches of remote sensing imagery. See [link to relevant documentation]. Figure 1 As shown, the method mainly includes the following steps:

[0039] Step S110: Obtain the preceding and following time-phase images to be detected, and slice the preceding and following time-phase images.

[0040] Step S120: Input the slice data into the pre-trained instance constraint change detection network to obtain the predicted change map containing the patches; wherein, the instance constraint change detection network includes a backbone network, a feature fusion module, an instance constraint decoding module and an instance constraint loss, the instance constraint loss is used to calculate the loss of each changed patch in the image during model training, and the losses of all changed patches are accumulated for gradient update;

[0041] Step S130: Perform small patch filtering and hole filling on the predicted change map, and simplify the outline of the changed patches in the predicted change map to obtain the patch optimization result;

[0042] Step S140: The optimization results of the patch are filled with a raster and converted to vector to generate the target change vector results.

[0043] For ease of understanding, the following provides a detailed description of the instance constraint change detection method for fragmented patches in remote sensing images provided in the embodiments of this application.

[0044] First, the instance constraint change detection network provided in the embodiments of this application will be described: the instance constraint change detection network includes a backbone network, a feature fusion module, an instance constraint decoding module, and an instance constraint loss, see [link to relevant documentation]. Figure 2 As shown below, each specific module will be explained in detail below.

[0045] I. Backbone Network

[0046] The backbone network is a swin-transformer structure used to extract image features at multiple scales. The sub-network structure used to extract features corresponding to each scale is composed of multiple stacked swin-blocks. The number of swin-blocks corresponding to the sub-network structure at different scales is not exactly the same. The swin-blocks adopt a moving window attention mechanism.

[0047] By employing the Shifted Window Multihead Self-Attention mechanism in swin-block, the information interaction between windows can be enhanced while reducing computational complexity, thus more effectively modeling the contextual information of the image.

[0048] In one specific implementation, the Swin-transformer extracts image features at four scales. The number of Swin-blocks for the four stages is set to 2, 2, 6, and 2 respectively. This configuration reduces the number of parameters while maintaining good feature extraction capabilities. The activation function for the Swin-block is set to Gelu, and LayerNorm is used for normalization, which helps the backbone network converge faster. The stride of the feature extraction modules in the four stages is set to 4, 2, 2, and 2 respectively. Taking a 512×512 input image as an example, the output feature map sizes are 128×128, 64×64, 32×32, and 16×16, with the feature map channels set to 96, 192, 384, and 768 respectively.

[0049] II. Feature Fusion Module

[0050] The scale of the feature fusion module is set to correspond to the scale of the backbone network. The feature fusion module is used to calculate the difference features by adding and averaging the feature maps of the previous and subsequent phases at different scales, and then using the convolution module to perform feature fusion after concatenating the difference feature maps obtained by the two calculation methods in the channel dimension.

[0051] In one implementation, the aforementioned feature fusion module is an Algebra feature fusion module. The Algebra feature fusion module calculates the difference features by adding and averaging the features from preceding and following phases at four different scales, and then subtracting them. The difference feature maps obtained from the two calculation methods are then concatenated along the channel dimension and further fused using a convolutional module. The Conv-Block convolutional module of the feature fusion module consists of Conv, BN, and ReLU operators connected in sequence. Based on the four scales of image features output by Swin, the feature fusion module is also divided into four scales. The input channels of the Conv operator are set to the sum of the number of channels in the preceding and following phase feature maps, and the output channels are equal to the number of channels in the preceding phase feature map. The kernel size is 1×1, and the stride is 1, ensuring that the two difference feature maps output higher-quality change features after convolutional fusion.

[0052] III. Instance Constraint Decoding Module

[0053] The construction idea of ​​the instance constraint decoding module is derived from instance segmentation. This embodiment introduces the concept of objects, transforming the traditional pixel-by-pixel classification change detection task into an object classification change detection task. It predicts the location and category of the mask to achieve the prediction of changed patches. The instance constraint decoding module includes: a pixel decoding module, a Transformer decoder module, a multilayer perceptron module, and an instance loss function; among which,

[0054] 1) The Pixel decoding module performs convolutional fusion on the differential feature maps at multiple scales output by the feature fusion module. The Pixel decoding module is an FPN (Feature-Focused Network) structure network that performs convolutional fusion on the feature maps at four scales output by the feature fusion module, improving the network's ability to learn multi-scale features. Normalization parameters are set using GN, and the activation function is ReLU.

[0055] 2) The Transformer decoder module performs masked feature lookup on the smallest highest-order feature map output by the feature fusion module. That is, the input for masked feature lookup is the smallest highest-order feature map output by the feature fusion module. The Transformer decoder module includes multiple network layers, each of which includes a self-attention network, a cross-attention network, and a feedforward network. The feature dimension of each network is preset with a corresponding target dimension.

[0056] In one example, the Transformer decoder module includes six network layers, each consisting of a self-attention network, a cross-attention network, and a feedforward network. The target dimension of each network can be set to 256.

[0057] 3) The Multi-Layer Perception (MLP) module includes a category feature submodule and a mask feature submodule. The category feature submodule is used to generate the final predicted category score and can be set to one MLP linear layer. The mask feature submodule is used to generate the final mask feature map and includes three MLP linear layers. The final category transformation map is generated by performing matrix multiplication on the predicted category score generated by the category feature submodule and the mask feature map generated by the mask feature submodule.

[0058] 4) The instance loss of the instance constraint decoding module includes classification loss and masking loss. The instance constraint loss of the instance constraint decoding module uses conventional CE, Dice, and Focal losses, focusing on pixel-level feature learning. The specific formula is shown below:

[0059] 4-1: Classification Loss

[0060]

[0061] Among them, z and z gt These refer to the prediction result and the ground truth label, respectively. N represents the number of ground truth masks, σ indicates that there is a matching correspondence between the prediction mask and the ground truth mask, and p σ(j) It is the class prediction value of the j-th prediction mask. The category value represents the truth mask.

[0062] 4-2: Masking Loss

[0063]

[0064] Where, parameters z, z gt The meanings of , N, and σ are the same as those described for classification loss, and λ dice Here are the loss weights for Dice loss, set to 1.0, m σ(j) This indicates that there exists a matching j-th prediction mask. Let λ represent the j-th truth mask. focal The loss weight coefficient for Focalloss is set to 20.

[0065] IV. Instance Constraint Loss

[0066] In one implementation, the instance constraint loss is added to the instance constraint decoding module. Unlike the pixel-focused loss described above, the instance constraint loss calculates the loss for each changing patch in the image separately, and then sums the losses of all patches to complete the gradient update. The instance constraint loss includes the changing instance loss and the background instance loss.

[0067] 1) Loss on changed instances:

[0068]

[0069] Where N is the number of changed instances in the truth labels, w pos γ is the loss adjustment coefficient for varying instances, and γ is the loss scaling factor. For the i-th instance of the changed truth value, p pos For predicting the probability of change, ε is a parameter used to prevent loss calculation anomalies.

[0070] 2) Background instance loss:

[0071]

[0072] Among them, w neg m is the background instance loss adjustment coefficient. gt_bg For background truth labels, p neg This is a probability map for the background.

[0073] The instance constraint decoding module mentioned above only participates during training and not during inference, thus not increasing the cost of inference. By employing this instance constraint decoding module, the instance constraint change detection method for fragmented patches described in this application has advantages over previous methods, including high accuracy, strong generalization performance, and fast inference speed. It ensures correct detection of changed patches while also considering their shape and contour information, solving problems such as broken, jagged, and discontinuous edges of changed patches.

[0074] For step S110 above, the image slicing process can first extract the image information corresponding to the previous and subsequent temporal images respectively. The image information includes at least image coordinates, number of rows and columns, and resolution. Based on the image information, the image intersection area of ​​the previous and subsequent temporal images is calculated. Within the image intersection area, slice index coordinates are established according to the sliding window size and sliding window step size corresponding to the images, so as to perform image slicing based on the slice index coordinates.

[0075] In one example, the sliding window size can be set to 512×512 with a step size of 384. Within the intersecting region, slice index coordinates are established based on the corresponding sliding window size and step size of the image. Before inputting the slices into the network, the higher-resolution sliding window slices are scaled using the Bilinear method.

[0076] Considering that there may be inconsistencies in resolution between preceding and following temporal images, in order to ensure processing accuracy, in one embodiment, when there are inconsistencies in resolution between preceding and following temporal images, the resolution scaling factor of the preceding and following temporal images can be calculated; if the resolution scaling factor is less than a preset threshold, the resolution scaling factor is optimized to obtain an optimized target scaling factor; based on the target scaling factor, the sliding window size and sliding window step size of the higher resolution image are enlarged, and the slice index target is reconstructed based on the enlarged sliding window size and sliding window step size.

[0077] Figure 3 A schematic diagram of slices of sequential and sequential remote sensing images at different resolutions is shown. Based on this schematic diagram, in one example, when the resolutions of sequential and sequential remote sensing images are different, the resolution scaling factor r of the sequential and sequential images can be calculated. If r is less than 1, then r = 1 / r. The sliding window size and step size of the lower resolution image remain unchanged, while the sliding window size and step size of the higher resolution image are both magnified using the resolution scaling factor (r = 1 / r).

[0078] This approach effectively addresses the issue of inconsistent ground features within the same sliding window size due to differences in temporal resolution between different phases in actual production environments, while also improving the accuracy of image preprocessing.

[0079] In step S120 above, the slice data is input into a pre-trained instance constraint change detection network for slice interpretation using the network model. The instance constraint change detection network InstMaskCDNet has completed its training and generated a weight file containing the optimal network parameters. When performing the interpretation task, the network used in this invention first loads the weight file to instantiate the model, then feeds the preceding and following time-phase slices into the instance constraint change detection network for change map prediction, outputting a binary change map, i.e., a predicted change map containing patches.

[0080] For step S130 above, see [link / reference]. Figure 4 As shown, a pre-set area filtering threshold for polygons can be set. All polygons in the predicted change map are then iterated through. Polygons with excessively small areas are removed by comparing their areas to the small polygon area filtering threshold. Then, hole localization and filling operations are performed on the filtered polygons to ensure the integrity of the change maps.

[0081] Furthermore, when simplifying the outline of changed features, see [reference needed]. Figure 5 As shown, a contour optimization area threshold can be set, all connected components of the transformed graph can be calculated, and the contour simplification process can be performed based on the comparison between the area of ​​the connected components and the contour optimization threshold. If the requirements are met, the contour of the region is simplified. In specific implementation, a contour extraction algorithm is used to obtain the contour of the transformed patch, morphological methods are used to eliminate jagged edges of the patch, a buffer is used to smooth the edge contour of the patch, then a minimum adjacent polygon is generated in the region to simplify the edge contour, and a third-order Bézier curve is used to smoothly fit the simplified edge contour point set. Finally, the transformed pixel values ​​are filled in the smoothed edge region to complete the patch contour optimization process.

[0082] For step S140 above, repeat steps S120 and S130 to complete all large image tile prediction and raster filling processing, and perform vector conversion operation on the final changed large image to generate the required large image change vector result.

[0083] Figure 6 The diagram illustrates a flowchart outlining the execution of a method according to an embodiment of this application. Compared to existing change detection algorithms, this method improves the accuracy and effectiveness of detecting changed ground features by using a pre-trained instance-constrained change detection network to perform change detection on both the preceding and following temporal images. Furthermore, by performing a series of processing steps on the patches (small patch filtering, hole filling, and contour simplification), the completeness and smoothness of the detected changed patches can be improved, significantly enhancing the accuracy of change detection.

[0084] Based on the above method embodiments, this application also provides an instance constraint change detection device for fragmented patches in remote sensing images, see [link to relevant documentation]. Figure 7 As shown, the device mainly includes the following parts:

[0085] The image slicing module 710 is used to acquire the previous and subsequent temporal images to be detected, and to slice the previous and subsequent temporal images.

[0086] The network detection module 720 is used to input the slice data into a pre-trained instance constraint change detection network to obtain a predicted change map containing patches. The instance constraint change detection network includes a backbone network, a feature fusion module, an instance constraint decoding module, and an instance constraint loss. The instance constraint loss is used to calculate the loss of each changed patch in the image during model training and to accumulate the losses of all changed patches for gradient update.

[0087] The patch processing module 730 is used to filter small patches and fill holes in the predicted change map, and to simplify the outline of the changed patches in the predicted change map to obtain the patch optimization result.

[0088] The vector conversion module 740 is used to perform raster filling and vector conversion on the optimization results of the patches to generate the target change vector results.

[0089] In one feasible implementation, the backbone network is a swin-transformer structure used to extract image features at multiple scales; the sub-network structure used to extract features corresponding to each scale is composed of multiple stacked swin-blocks, and the number of swin-blocks corresponding to the sub-network structure at different scales is not exactly the same, and the swin-blocks adopt a moving window attention mechanism; the scale of the feature fusion module is set to correspond to the scale of the backbone network, and the feature fusion module is used to calculate the difference features by adding and averaging the feature maps of the previous and subsequent phases at different scales and by subtracting them, and then concatenating the difference feature maps obtained by the two calculation methods in the channel dimension and using the convolution module to perform feature fusion.

[0090] In one feasible implementation, the instance constraint decoding module includes: a pixel decoding module, a Transformer decoder module, a multilayer perceptron module, and an instance loss; wherein, the pixel decoding module performs convolutional fusion on the difference feature maps of multiple scales output by the feature fusion module; the Transformer decoder module performs mask feature query on the smallest highest-order feature map output by the feature fusion module, the Transformer decoder module includes multiple network layers, each network layer includes a self-attention network, a cross-attention network, and a feedforward network, and the feature dimension of each network is preset with a corresponding target dimension; the multilayer perceptron module includes a category feature submodule and a mask feature submodule, the category feature submodule is used to generate the final predicted category score, and the mask feature submodule is used to generate the final mask feature map; the final category change map is generated by performing matrix multiplication on the predicted category score generated by the category feature submodule and the mask feature map generated by the mask feature submodule; the instance loss of the instance constraint decoding module includes classification loss and mask loss.

[0091] In one feasible implementation, the instance constraint loss includes the change instance loss:

[0092]

[0093] Where N is the number of changed instances in the truth labels, w pos γ is the loss adjustment coefficient for varying instances, and γ is the loss scaling factor. For the i-th instance of the changed truth value, p pos For predicting the probability of change, ε is a parameter used to prevent loss calculation anomalies.

[0094] In one feasible implementation, the instance constraint loss includes the background instance loss:

[0095]

[0096] Among them, w neg m is the background instance loss adjustment coefficient. gt_bg For background truth labels, p neg This is a probability map for the background.

[0097] In one feasible implementation, the image slicing module 710 is further configured to:

[0098] Extract image information corresponding to the preceding and following time-phase images, including at least image coordinates, number of rows and columns, and resolution; calculate the image intersection region between the preceding and following time-phase images based on the image information; establish slice index coordinates within the image intersection region according to the corresponding sliding window size and sliding window step size, and perform image slicing based on the slice index coordinates.

[0099] In one feasible embodiment, the above apparatus further includes: a resolution processing module, used for:

[0100] Calculate the resolution scaling factor of the previous and subsequent time-phase images; if the resolution scaling factor is less than a preset threshold, perform preset optimization processing on the resolution scaling factor to obtain the optimized target scaling factor; based on the target scaling factor, enlarge the sliding window size and sliding window step size of the higher resolution image, and reconstruct the slice index target based on the enlarged sliding window size and sliding window step size.

[0101] The instance constraint change detection device for fragmented patches in remote sensing images provided in this application has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the embodiment of the instance constraint change detection device for fragmented patches in remote sensing images can be referred to the corresponding content in the aforementioned embodiment of the method for detecting instance constraint change of fragmented patches in remote sensing images.

[0102] This application also provides an electronic device, such as... Figure 8 The diagram shows the structure of the electronic device 100, which includes a processor 81 and a memory 80. The memory 80 stores computer-executable instructions that can be executed by the processor 81. The processor 81 executes the computer-executable instructions to implement any of the above-mentioned methods for detecting instance constraint changes in fragmented patches of remote sensing images.

[0103] exist Figure 8 In the illustrated embodiment, the electronic device further includes a bus 82 and a communication interface 83, wherein the processor 81, the communication interface 83, and the memory 80 are connected via the bus 82.

[0104] The memory 80 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 83 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 82 may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 82 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0105] Processor 81 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 81 or by software instructions. The processor 81 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory. The processor 81 reads the information in the memory and, in conjunction with its hardware, completes the steps of the instance constraint change detection method for broken patches in remote sensing images as described in the aforementioned embodiment.

[0106] This application also provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are called and executed by a processor, they cause the processor to implement the above-described method for detecting instance constraint changes in broken patches of remote sensing images. For specific implementation details, please refer to the foregoing method embodiments, which will not be repeated here.

[0107] The computer program product of the method and apparatus for detecting instance constraint changes of broken patches in remote sensing images provided in this application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0108] Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of this application.

[0109] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0110] In the description of this application, it should also be noted that, unless otherwise expressly specified and limited, the terms "set up," "connected," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for instance-constrained change detection for broken map patches of remote sensing images, characterized in that, The method comprises the following steps: acquire the pre-time image and post-time image to be detected, and perform image slicing on the pre-time image and post-time image; input the sliced data into a pre-trained instance constraint change detection network to obtain a predicted change map containing a graph spot; wherein the instance constraint change detection network comprises a backbone network, a feature fusion module, an instance constraint decoding module and an instance constraint loss, the instance constraint loss is used to calculate the loss of each change graph spot in the image during model training, and the loss of all change graph spots is accumulated for gradient update; perform small spot filtering and hole filling processing on the predicted change map, and perform contour simplification on the change graph spot in the predicted change map to obtain a graph spot optimization result; perform raster filling on the graph spot optimization result and perform vector conversion to generate a target change vector result; wherein the instance constraint decoding module comprises a Pixel decoding module, a Transformer decoder module, a multi-layer perception module and an instance loss; wherein the Pixel decoding module performs convolution fusion on the difference feature maps of multiple scales output by the feature fusion module; the Transformer decoder module performs mask feature query on the highest-order feature map with the smallest size output by the feature fusion module, the Transformer decoder module comprises multiple network layers, each network layer comprises a self-attention network, a cross-attention network and a feedforward network, and the feature dimension of each network is pre-set to a corresponding target dimension; the multi-layer perception module comprises a category feature submodule and a mask feature submodule, the category feature submodule is used to generate a final predicted category score, and the mask feature submodule is used to generate a final mask feature map; a final category change map is generated by performing matrix multiplication processing on the predicted category score generated by the category feature submodule and the mask feature map generated by the mask feature submodule; the instance loss of the instance constraint decoding module comprises a classification loss and a mask loss.

2. The method of claim 1, wherein the method further comprises: The backbone network is a swin-transformer structure, which is used to extract image features of multiple scales; the subnetwork structure for extracting features of each scale is stacked by multiple swin-blocks, the number of swin-blocks corresponding to the subnetwork structures of different scales is not completely the same, and the swin-block adopts a moving window attention mechanism; the scale of the feature fusion module corresponds to the scale of the backbone network, and the feature fusion module is used to add and subtract the pre-time feature map and the post-time feature map under different scales to calculate difference features, and then perform feature fusion on the difference feature maps obtained by the two calculation methods after splicing in the channel dimension using a convolution module.

3. The method of claim 1, wherein the method further comprises: The instance constraint loss comprises a change instance loss: where N is the number of change instances in the ground truth label, is a change instance loss adjustment coefficient, is a loss scaling factor, is the i-th change ground truth instance, is the predicted change probability map, is a parameter to prevent loss calculation abnormalities.

4. The method of claim 1, wherein the method further comprises: The instance constraint loss comprises a background instance loss: wherein, is a background instance loss adjustment coefficient, is a background ground truth label, is a background prediction probability map.

5. The method of claim 1, wherein the method further comprises: perform image slicing on the pre-time image and post-time image, which comprises: extract image information corresponding to the pre-time image and post-time image respectively, the image information at least comprises image coordinates, row and column numbers and resolution; Calculate an image intersection region of the pre-phase image and the post-phase image based on the image information; Establish slice index coordinates in the image intersection region according to a sliding window size and a sliding window step corresponding to the images, so as to perform image slicing based on the slice index coordinates.

6. The method of claim 5, wherein the method further comprises: The method further comprises: Calculate a resolution ratio factor of the pre-phase image and the post-phase image; If the resolution ratio factor is less than a preset threshold, perform preset optimization processing on the resolution ratio factor to obtain an optimized target ratio factor; Enlarge the sliding window size and the sliding window step of the image with higher resolution based on the target ratio factor, so as to reconstruct slice index targets based on the enlarged sliding window size and the sliding window step.

7. A device for detecting changes in instances of fragmented map patches in remote sensing imagery, characterized in that Comprise: An image slicing module, configured to acquire a pre-phase image and a post-phase image to be detected, and perform image slicing on the pre-phase image and the post-phase image; A network detection module, configured to input slice data into a pre-trained instance constraint change detection network to obtain a predicted change map containing patches; wherein the instance constraint change detection network comprises a backbone network, a feature fusion module, an instance constraint decoding module, and an instance constraint loss, the instance constraint loss is used to calculate the loss of each change patch in the image during model training, and the loss of all change patches is accumulated for gradient update; A patch processing module, configured to perform small patch filtering and hole filling processing on the predicted change map, and perform contour simplification on the change patches in the predicted change map to obtain a patch optimization result; A vector conversion module, configured to perform raster filling and vector conversion on the patch optimization result to generate a target change vector result; The instance constraint decoding module comprises a Pixel decoding module, a Transformer decoder module, a multi-layer perception module, and an instance loss; wherein The Pixel decoding module performs convolution fusion on the difference feature maps of multiple scales output by the feature fusion module; The Transformer decoder module performs mask feature query on the highest-order feature map with the smallest size output by the feature fusion module, and the Transformer decoder module comprises multiple network layers, each network layer comprises a self-attention network, a cross-attention network, and a feedforward network, and the feature dimension of each network is preset to a corresponding target dimension; The multi-layer perception module comprises a class feature submodule and a mask feature submodule, the class feature submodule is used to generate a final predicted class score, and the mask feature submodule is used to generate a final mask feature map; a final class change map is generated by performing matrix multiplication processing on the predicted class score generated by the class feature submodule and the mask feature map generated by the mask feature submodule; The instance loss of the instance constraint decoding module comprises a classification loss and a mask loss.

8. An electronic device, comprising: Comprise a processor and a memory, the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the instance constraint change detection method for broken patches of remote sensing images in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions, when the computer executable instructions are invoked and executed by the processor, the computer executable instructions cause the processor to implement the instance constraint change detection method for the broken map patches of remote sensing images according to any one of claims 1 to 6.

Images