Remote sensing change detection method and system based on deep reinforcement learning

Abstract

The invention provides a remote sensing change detection method and system based on deep reinforcement learning, and the method comprises the steps: carrying out the feature extraction and difference modeling of an input multi-temporal remote sensing image through a deep learning network, so as to generate a preliminary change prediction map; and introducing a reinforcement learning module, taking the preliminary change prediction map as an initial state, and performing multi-round iterative optimization on the preliminary change prediction map through a decision process comprising a state space, an action space and a reward function so as to generate a final change detection result. And the reward function comprehensively considers an intersection-to-union ratio improvement value, an F1 score improvement value and cross entropy loss, and introduces a misclassification penalty enhancement mechanism to preferentially repair missing report and false alarm areas. According to the method, through iterative optimization of reinforcement learning, errors in a preliminary detection result can be effectively corrected, and the detection precision and robustness are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and more specifically, to a remote sensing change detection method and system based on deep reinforcement learning. Background Technology

[0002] Remote sensing change detection technology, as an important means of identifying land cover or land use changes using multi-temporal remote sensing imagery, is widely used in fields such as urban expansion monitoring, ecological environment assessment, disaster response, and resource management. With the improvement of remote sensing image resolution and the shortening of observation cycles, land cover changes exhibit more detailed and temporally complex characteristics. Especially when facing complex backgrounds, subtle changes, and irregular boundaries, traditional change detection methods have gradually revealed problems such as low accuracy and poor generalization ability. Traditional methods typically rely on fixed feature extraction and judgment rules, making it difficult to adapt to various heterogeneous data and complex environments, especially posing significant challenges in the practical application of high-precision remote sensing change detection.

[0003] Currently, deep learning-based change detection methods, especially Convolutional Neural Networks (CNNs), U-Net, and Siamese networks, have made significant progress in pixel-level semantic segmentation tasks. These methods can automatically extract features from images and perform end-to-end change region extraction. For example, existing technologies often use the construction of Siamese network structures to effectively extract depth features from multi-temporal images and perform difference comparisons, thereby achieving preliminary localization of change regions. However, these deep learning-based methods still face a series of challenges in practical applications. First, because convolution and pooling operations in the network inevitably lose spatial detail information of the image, the model's boundary segmentation of change regions is not refined enough, and the final detection results often have problems such as blurred edges or jagged edges, affecting detection accuracy. Second, remote sensing images are highly susceptible to interference from non-real changes in ground features, such as changes in illumination, seasonal changes, and cloud cover. These factors can create "pseudo-changes" in images, and existing methods are not robust to this type of noise, easily generating false alarms, that is, misclassifying unchanged areas as changed areas. Furthermore, existing models exhibit significantly reduced recognition capabilities when faced with small-scale changes, morphological variations in non-rigid targets (such as rivers and lakes), and complex backgrounds, leading to false negatives and the omission of truly changed areas. Finally, most existing methods employ a one-time forward propagation "end-to-end" model, directly outputting detection results without the iterative optimization process of repeated review and correction by human experts. This makes it difficult to fine-tune and correct problematic or erroneous areas in the initial detection results. In summary, these methods rely on static training, lack interaction with the external environment, and cannot dynamically adjust prediction strategies. Especially when faced with ambiguous boundaries, spurious changes, or heterogeneous data, the models are prone to false positives and false negatives, limiting their stability and universality in practical remote sensing applications.

[0004] To enhance the ability of models to detect changes in complex scenes, reinforcement learning methods have been introduced in recent years. Reinforcement learning, by constructing an interaction mechanism between the agent and the environment, can continuously optimize decision-making strategies based on feedback rewards, breaking the limitations of traditional deep learning models' "static prediction – single supervision." Deep Reinforcement Learning (DRL) combines the advantages of reinforcement learning and deep neural networks, exhibiting powerful dynamic learning and policy transfer capabilities, especially in tasks such as image segmentation, path planning, and intelligent decision-making.

[0005] Although deep reinforcement learning has made significant progress in image processing and remote sensing, research on its effective application in remote sensing change detection is still in the exploratory stage. Existing research faces several pressing issues, including unreasonable task modeling, insufficient state space design, and a lack of targeted reward mechanisms. These factors lead to low model learning efficiency, poor policy convergence, and difficulty in effectively identifying boundary changes, missing details, and cross-temporal and spatial variation patterns.

[0006] Therefore, there is a market need for a remote sensing change detection method and system that can improve the recognition accuracy and generalization ability of complex and changing regions by combining deep feature extraction and reinforcement learning strategy optimization. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the purpose of this invention is to provide a remote sensing change detection method and system based on deep reinforcement learning, in order to solve the problems of imprecise change boundary segmentation, poor robustness to false changes, high false alarm and missed detection rates in complex scenarios, and lack of iterative optimization of detection results in existing technologies.

[0008] A remote sensing change detection method based on deep reinforcement learning, provided by the present invention, includes: A deep learning network is used to extract multi-scale deep features and compare differences between paired multi-temporal remote sensing images to generate a preliminary change prediction map and a multi-scale feature pyramid. The preliminary change prediction map represents the probability of changes in the position of each pixel, and the multi-scale feature pyramid contains feature maps with contextual information extracted by different network depths. A reinforcement learning module is introduced, which takes the preliminary change prediction map and the multi-scale feature pyramid as the initial state. Through a decision-making process that includes a state space, an action space and a reward function, the preliminary change prediction map is iteratively optimized in multiple rounds to generate the final change detection result.

[0009] Preferably, the step of generating a preliminary change prediction map using a deep learning network includes: A twin network structure is adopted, wherein the twin network includes an encoder with shared weights, and the encoder extracts multi-scale features of the multi-temporal remote sensing images respectively; Feature difference modeling is performed on the extracted multi-scale features to obtain differential features that reflect image changes; The difference features are input into the decoder, which performs upsampling and feature fusion processing to output the preliminary change prediction map.

[0010] Preferably, the input state of the reinforcement learning module consists of three parts: The deep feature pyramid is extracted by the encoder of the deep learning network and processed by the multi-scale feature fusion module; The segmentation prediction results of the previous iteration, wherein the preliminary change prediction map was used as the segmentation prediction results in the first iteration; The difference map between the previous prediction results and the true labels is used to guide the policy network to focus on the boundary of change and difficult areas.

[0011] Preferably, the reinforcement learning module guides the iterative optimization process through a reward function R; The reward function R is defined as follows:

[0012] in, and and represent the intersection-union ratio and F1 score after performing the optimization actions, respectively. and Then, these are the cross-union ratio and F1 score before performing the optimization action, CrossEntropyLoss is the cross-entropy loss term, and α, β, and γ are preset weight hyperparameters.

[0013] Preferably, the reward function introduces a misclassification penalty enhancement mechanism, which assigns different weights to misclassified pixels through a location weight map, with the penalty weight of pixels in the missed detection area and false alarm area being greater than that of correctly classified pixels; The underreported area is the area that actually changed but was predicted not to have changed; The false alarm area is the area that is actually unchanged but is predicted to change.

[0014] Preferably, a phased training strategy is adopted, specifically including: The first step is to train the deep learning network independently until the model converges; The second step is to freeze the encoder parameters of the deep learning network and start training the policy network in the reinforcement learning module. The third step is to perform joint fine-tuning on the entire model to improve its overall performance.

[0015] A remote sensing change detection system based on deep reinforcement learning, according to the present invention, includes: The data preprocessing module is used to normalize and unify the size of the input multi-temporal remote sensing images; The deep learning prediction module, with a built-in Siamese network structure, is used to extract image features and generate preliminary change prediction maps. The reinforcement learning optimization module is used to iteratively optimize the preliminary change prediction map and output the final change detection result. The storage module is used to store the trained model parameters and remote sensing dataset; The processing unit is used to execute the computational tasks of each module.

[0016] Preferably, the reinforcement learning optimization module includes: State building unit, used to build input state containing deep features, previous prediction results and difference map; The multi-scale feature fusion unit is used to fuse and refine the multi-scale features output by the deep learning prediction module. A policy network is used to output pixel-level classification adjustment actions based on the input state; The reward calculation unit is used to calculate the reward value according to the preset reward function to guide the training of the policy network.

[0017] Preferably, the normalization operation of the data preprocessing module adopts the mean-standard deviation normalization method, and the size unification operation is achieved by interpolation to ensure consistent resolution of multi-temporal remote sensing images.

[0018] Preferably, the decoder of the deep learning prediction module uses a transposed convolutional layer to achieve upsampling, and fuses multi-scale features output by the encoder through skip connections to improve the detailed representation capability of the initial change prediction map.

[0019] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention is the first to organically integrate deep learning and reinforcement learning in the field of change detection, constructing a change detection framework with a dual-branch structure of "deep feature extraction + policy optimization". The deep learning branch utilizes convolutional neural networks (such as Siamese U-Net) to extract multi-scale spatial features from multi-temporal remote sensing images and generate preliminary change area predictions. The reinforcement learning branch then uses these predictions as a basis to dynamically correct and finely optimize the segmentation results through a policy iterative learning mechanism. This fusion of deep learning's feature extraction capabilities and reinforcement learning's policy optimization mechanism effectively enhances the model's ability to identify complex changes in ground features.

[0020] 2. This invention constructs a reward function for comprehensive performance improvement. This reward function comprehensively considers the IoU improvement value, F1 score improvement value, and cross-entropy loss term for changing region identification, and is used to comprehensively measure the degree of improvement in prediction accuracy before and after policy optimization. At the same time, a misclassification penalty enhancement mechanism is introduced, which assigns higher weights to key misclassified regions such as "false alarms" and "missed alarms", guiding the policy network to focus on error-prone regions and improving the model's sensitivity and repair capability to fine-grained changes.

[0021] 3. This invention constructs a reinforcement learning policy optimization mechanism adapted to remote sensing image features by designing a state space with semantic feature input, historical prediction feedback, and error prompts. The policy network is trained using Proximal Policy Optimization (PPO), which has the advantages of fast convergence and high stability, and can gradually improve the clarity of segmentation boundaries and prediction accuracy during iteration. Through a state-action-reward cyclic update method, adaptive correction of prediction results is achieved, improving the accuracy of changing region identification, especially in robustness to scenarios with blurred boundaries, frequent false changes, or heterogeneous data.

[0022] 4. This method supports end-to-end joint training, adapts to various remote sensing data sources and deep network structures, and has good scalability. The reinforcement learning module can be flexibly inserted into existing change detection architectures to enhance existing models through policy improvements, thereby increasing the model's adaptability and performance. Attached Figure Description

[0023] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating the remote sensing change detection method based on deep reinforcement learning according to the present invention. Figure 2 The model structure diagram proposed in this invention; Figure 3 This is a schematic diagram of the reinforcement learning training and update process in this invention. Detailed Implementation

[0024] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0025] Example 1 This invention provides a remote sensing change detection method based on deep reinforcement learning. The method aims to significantly improve the accuracy and robustness of change detection, particularly in handling detailed boundary conditions and complex scenarios, through a two-stage optimization framework. First, a deep learning network is used to generate preliminary change detection results. Then, a reinforcement learning agent is introduced to iteratively refine these results. This two-stage optimization framework includes a deep learning prediction stage and a reinforcement learning optimization stage.

[0026] In the deep learning prediction stage, a pair of precisely registered remote sensing images acquired at different time points are first received: time-phase image T1 and time-phase image T2. These two images are simultaneously input into the deep learning prediction module, which performs preliminary feature extraction and change analysis. As a preferred approach, the deep learning prediction module employs a Siamese network structure, such as Siamese U-Net, which can effectively extract multi-scale depth features from the two input images and compare the differences between these features to generate a preliminary change prediction map. This prediction map can be understood as a coarse mask of the changed region, where the grayscale value of each pixel represents the probability of a change at that location. Simultaneously, during feature extraction, the deep learning prediction module also generates a multi-scale feature pyramid. This pyramid contains feature maps extracted from different network depths, rich in contextual information, providing crucial semantic information for subsequent optimization steps.

[0027] Subsequently, the system enters the reinforcement learning optimization phase. The initial change prediction map and multi-scale feature pyramid generated by the deep learning prediction phase serve as core inputs to the reinforcement learning optimization module. This module simulates the decision-making process of human experts repeatedly reviewing and progressively revising images. Specifically, the module performs multi-round iterative policy optimization on the initial change prediction map through a reinforcement learning loop that includes state, action, and reward. In each iteration, an optimal adjustment action is determined based on the current state (including deep features and the current prediction map) to correct potential errors in the prediction map, such as blurred boundaries, false changes, or missed small targets. During the training phase, the system uses real labeled data to calculate a reward signal to guide the learning direction of the policy network, enabling it to learn to make adjustments that maximize detection accuracy. After multiple rounds of optimization, the reinforcement learning optimization module finally outputs a more accurate final change detection result.

[0028] The following will be combined with the appendix Figure 1 and 2 The key steps and modules in this method are described in detail, such as... Figure 1 As shown, the method includes: In terms of data preparation and preprocessing, the data type consists of multi-temporal remote sensing image pairs with ground truth change labels. To ensure the effectiveness and generalization ability of the model training, this embodiment selects several industry-recognized publicly available remote sensing change detection datasets, including but not limited to LEVIR-CD, DSIFN, WHU-CD, CDD, and SYSU-CD. Before inputting the image data into the model, standard preprocessing is required. Specifically, the standard preprocessing includes: size unification, i.e., cropping or scaling all images to a fixed size (e.g., 256x256 pixels or 512x512 pixels) to adapt to the network input requirements; data normalization, i.e., linearly scaling pixel values ​​from, for example, the range of 0-255 to the range of 0-1 or -1-1, which helps to accelerate model convergence and improve training stability; and data augmentation, i.e., to expand the number of training samples and prevent model overfitting, various online data augmentation techniques can be used, such as random rotation, horizontal or vertical flipping, random cropping, and color dithering (i.e., slightly adjusting brightness, contrast, and saturation).

[0029] Regarding the deep learning prediction stage, this embodiment employs a Siamese U-Net structure to extract features from paired multi-temporal remote sensing images, constructing a multi-scale feature pyramid to characterize the spatiotemporal changes of ground features at different times. The deep features extracted by the encoder are modeled through feature differences and then input into the decoder, outputting a preliminary change prediction map, providing basic segmentation results for subsequent reinforcement learning optimization. Figure 2 As shown, the specific implementation of the deep learning prediction stage is described below: The deep learning prediction module preferably employs a Siamese U-Net architecture. The T1 and T2 temporal images are fed into two structurally identical encoders with shared weights. This shared weight design ensures that the same feature extraction criteria are used for both images, making subsequent feature comparisons more meaningful. The encoder typically consists of a series of convolutional blocks and downsampling layers (such as max-pooling layers), for example, the first half of VGG-16 or ResNet-34 can be used as the encoder backbone. As the network depth increases, the encoder can extract features layer by layer, from low-level texture and edge information to high-level semantic concepts. At each downsampling layer, a feature map of a specific scale is generated. These feature map sets from different levels together form a multi-scale feature pyramid, which is then passed to the subsequent reinforcement learning module.

[0030] At the deepest layer of the encoding path, the feature maps from the two time phases contain the richest semantic information. These two sets of feature maps are fed into a feature difference modeling module, which calculates the differences between the two time phase features to highlight the areas of change. A simple implementation is to subtract element-wise or calculate the absolute difference. A more efficient implementation is to concatenate the two sets of feature maps along the channel dimension, and then learn through one or more 1x1 convolutional layers, allowing the network to determine how to most effectively fuse and extract the difference information.

[0031] The output of the feature difference modeling module, i.e., the difference features, is then fed into the decoder. The decoder's structure is symmetrical to the encoder, consisting of a series of upsampling layers (such as transposed convolutions or bilinear interpolations) and convolutional blocks. Its purpose is to progressively restore the highly abstract semantic difference feature map to the resolution of the original input image. To address the spatial detail lost during downsampling, a key design feature of the U-Net architecture is the introduction of skip connections. Specifically, at each upsampling stage of the decoder, its feature map is concatenated and fused with the feature map from the corresponding layer of the encoder. For example, a 64x64 resolution feature map in the decoder is fused with a 64x64 resolution feature map in the encoder. This design allows the decoder to simultaneously utilize deep semantic information (from the difference features) and shallow spatial detail information (from skip connections), thereby generating predictions with clearer boundaries. Finally, the last layer of the decoder is typically a 1x1 convolutional layer followed by a sigmoid activation function, outputting a single-channel preliminary change prediction map with pixel values ​​between 0 and 1, representing the probability of change.

[0032] Regarding the reinforcement learning optimization stage, based on the preliminary change prediction map and multi-scale feature pyramid output from the deep learning prediction stage, a reinforcement learning optimization strategy is introduced to dynamically iteratively optimize the model, thereby further improving the accuracy and robustness of change region identification. For example... Figure 3 As shown, the goal of the reinforcement learning optimization phase is to refine the initial change prediction map. This process can be modeled as a Markov decision process, and its key elements are defined as follows: 1. State Space: The state is the basis for the decision-making of the reinforcement learning optimization module. In this invention, a state S_t (at the t-th iteration) is designed as a multi-channel tensor, composed of three stacked information parts: The first is the multi-scale feature pyramid. This part of the information originates from the encoder output of the deep learning prediction module and is processed by a dedicated multi-scale feature fusion module. It provides the reinforcement learning optimization module with rich, original multi-scale contextual information about the image content, which helps the reinforcement learning optimization module understand the specific context of changes, such as distinguishing between the demolition of buildings and seasonal vegetation changes.

[0033] The second is the prediction result of the previous round, that is, the prediction map generated by the agent after the (t-1)th iteration (in the first iteration, that is, at t=1, this is the preliminary change prediction map).

[0034] The third is the difference map. During the training phase, this is an error map generated by comparing the prediction results of the previous round with the true labels, which serves as feedback to guide the strategy to focus on key areas.

[0035] 2. Policy Network: The core of the agent is a policy network, typically a lightweight convolutional neural network. It receives a tensor consisting of a state space as input and outputs an action. For example, the network can consist of several residual blocks to effectively handle complex input states and output an action graph of the same size as the input prediction graph.

[0036] 3. Action Space: An action is an operation performed by the agent to modify the current prediction map. In this embodiment, an action is defined as an adjustment map of the same size as the prediction map. Each pixel value in this map represents the amount of adjustment to the classification probability of the corresponding pixel in the current prediction map. For example, an action could be a residual map added to the logistic values ​​of the current prediction map, where positive values ​​indicate an increase in the confidence level of the pixel and negative values ​​indicate a decrease.

[0037] 4. Environment: The environment receives actions from the agent and updates its state accordingly. Specifically, the environment is the current prediction graph. When an action is applied to the environment, a new, modified prediction graph is generated, which is the prediction result for the next round.

[0038] 5. Reward Function: The reward function is crucial for guiding the learning of the policy network. To make the optimization process more targeted, this invention designs a composite reward function, the mathematical expression of which is:

[0039] in, and and represent the intersection-union ratio and F1 score after performing the optimization actions, respectively. and These are the corresponding metrics before the optimization actions are performed. The first two items directly reward actions that improve core segmentation metrics, representing a highly efficient and intensive reward system. It is the cross-entropy loss between the new prediction map and the true label. As a penalty term, it can prevent the policy network from making a worse global prediction in pursuit of local IoU or F1 score improvement. , , These are preset weight hyperparameters, which need to be tuned through cross-validation. For example, they can be set to... This function is used to balance the importance of different reward components. It quantifies the actual improvement in detection performance caused by policy adjustments, taking into account both global accuracy and local error control.

[0040] The cross-entropy loss function measures the pixel-level error between the current predicted image and the true label, and its expression as a penalty term is:

[0041] Where: N is the number of samples. It is the true value of the i-th sample. It is the predicted value of the i-th sample.

[0042] Overall accuracy (OA) is defined as:

[0043] Where TP is the number of true positive cases, TN is the number of true negative cases, FP is the number of false positive cases, and FN is the number of false negative cases.

[0044] The expression for the F1 score is:

[0045] in, .

[0046] To further address the issues of spurious changes and missed detections, this invention introduces a misclassification penalty reinforcement mechanism into the reward function. This mechanism is implemented through a location weight map. When calculating the cross-entropy loss, pixels that are misclassified are assigned different weights based on their error type. Specifically, pixels in missed detection regions (those that are actually changed but predicted as unchanged) and false alarm regions (those that are actually unchanged but predicted as changed) are given significantly higher penalty weights than correctly classified pixels (for example, the loss weight for these critical error regions can be set to 5 or 10, while other regions are set to 1). This makes the reward signal more sensitive to correcting these two types of critical errors, thereby guiding the policy network to prioritize learning how to suppress spurious changes and recover missed small targets.

[0047] The entire reinforcement learning optimization process is an iterative loop: at step t, the policy network observes the state and outputs an action; the action acts on the environment, generating a new prediction graph and reward; this reward is used to update the parameters of the policy network, enabling it to make better decisions in the future. This loop is repeated a fixed number of times (e.g., 3 to 5 times), or until the prediction results converge. In this embodiment, the Proximal Policy Optimization (PPO) algorithm can be used to train the policy network. This algorithm introduces a pruning objective function to limit the step size of each update, thereby ensuring the stability of the training process and the efficiency of sample utilization.

[0048] To provide high-quality input states for reinforcement learning agents, this invention also designs a dedicated multi-scale feature fusion module. This module processes multi-scale feature inputs extracted from the deep learning network encoder. First, feature maps from different scales (which may have different numbers of channels) are passed through a parallel one-dimensional convolution module. This module uses multiple parallel 1x1 convolution kernels to unify the number of channels of all feature maps to a fixed value (e.g., 256). Next, these channel-unified feature maps are fed into a serial channel-spatial attention module. This module first weights the feature channels through a channel attention submodule to enhance features that are more discriminative for change detection tasks; then, the channel-weighted feature maps are passed through a spatial attention submodule to enhance spatial regions in the image that are highly correlated with changes. After refinement by the attention mechanism, the feature maps enter the upsampling and stitching unit. Here, all feature maps from lower resolutions are upsampled to the same resolution as the original input (or a certain target) through bilinear interpolation or other methods, and stitched along the channel dimension to form a high-dimensional feature tensor. Finally, the concatenated tensor undergoes deep information fusion and feature smoothing through a feature fusion and refinement unit (usually consisting of two to three 3x3 convolutional layers), ultimately outputting a high-resolution feature map as the deep feature pyramid part in the state space.

[0049] Regarding the model's training strategy and inference deployment, this invention employs a phased training strategy to ensure overall performance and training efficiency. In the first phase, the deep learning prediction module is trained independently. A standard loss function (such as binary cross-entropy loss or Dice loss) can be used, along with an optimizer like Adam, to train the Siamese U-Net end-to-end until its performance on the validation set converges. The goal of this phase is to enable the network to learn to extract effective spatiotemporal features and provide a reasonable initial prediction. In the second phase, the parameters of the encoder in the deep learning prediction module are frozen to maintain its feature extraction capability, and then the policy network in the reinforcement learning optimization module is trained. At this point, a pre-trained encoder is used to provide a fixed deep feature pyramid for the reinforcement learning process. In the third phase, after the policy network training is stable, the entire model (including the deep learning module and the reinforcement learning module) can be selectively fine-tuned end-to-end using a small learning rate, which helps the two modules work together better.

[0050] In the inference and deployment phase, the process is more straightforward. After inputting a pair of T1 and T2 temporal images into the system, a forward propagation is first performed through a pre-trained deep learning prediction module to obtain a preliminary change prediction map and a multi-scale feature pyramid. Then, a reinforcement learning optimization module takes over, performing a fixed number of iterative optimizations (e.g., 5). During this process, since there are no real labels, the difference map portion in the state space is ignored. After multiple rounds of optimization, the final prediction map is the final change detection result. The entire inference process requires no additional manual annotation, exhibiting a high degree of automation and deployment efficiency.

[0051] Example 2 This invention also provides a remote sensing change detection system based on deep reinforcement learning, configured to perform all the method steps described in Example 1. In one embodiment of this invention, the system can be deployed on a hardware platform with powerful computing capabilities, such as a server or workstation equipped with a high-performance graphics processing unit. The remote sensing change detection system can be logically divided into multiple functional modules.

[0052] The data preprocessing module is responsible for performing the data preparation and preprocessing operations described in Example 1. Upon receiving the input T1 and T2 time-phase images, this module can automatically perform normalization, size unification, and possible format conversion operations to ensure that the input data meets the requirements of subsequent processing modules. This module can be implemented by a series of software scripts or functions.

[0053] The deep learning prediction module serves as the core of the first stage of the system. Internally, it incorporates the Siamese network structure described in Example 1, such as the Siamese U-Net. This module receives preprocessed image pairs and, through its built-in shared-weight encoder, feature difference modeling module, and decoder, performs a series of calculations including feature extraction, difference modeling, and upsampling fusion, ultimately outputting a preliminary change prediction map and a multi-scale feature pyramid. It should be noted that the parameters of this module are fixed after training with a large amount of data.

[0054] The reinforcement learning optimization module, as the core of the second stage of the system, is responsible for refining the initial results. This module can be further subdivided into several cooperating sub-units: 1. A state construction unit, responsible for integrating deep features from the deep learning prediction module, the prediction results from the previous round, and the difference map generated during training (compared to the true labels) at the beginning of each iteration to construct a state space tensor that meets the required format. 2. A multi-scale feature fusion unit, whose structure and function are as described in Example 1, is specifically used to process and refine the multi-scale feature pyramid, providing high-quality deep feature input to the state construction unit. 3. A policy network, an independent, pre-trained neural network model that receives the output of the state construction unit and generates pixel-level classification adjustment actions based on the current policy. 4. A reward calculation unit, activated only during the training phase. Based on a preset reward function (including IoU boost, F1 score boost, and weighted cross-entropy loss), it calculates the reward value obtained for the current action and feeds this reward signal back to the training algorithm (such as a proximal policy optimization algorithm) to guide the parameter updates of the policy network.

[0055] The storage module can be composed of physical storage media such as hard drives, solid-state drives, or memory. It is used to persistently store various data required for the operation of the system, including but not limited to: remote sensing datasets used for training and testing, model parameters of trained deep learning prediction modules, model parameters of policy networks in trained reinforcement learning optimization modules, as well as intermediate results and final change detection results generated during processing.

[0056] The processing unit, as the computational core of the entire system, typically consists of a central processing unit (CPU) and at least one graphics processing unit (GPU). The CPU is responsible for scheduling the entire process, loading data, and handling some logic control, while the GPU, with its powerful parallel computing capabilities, is responsible for executing the intensive forward and backward propagation computations of the neural networks in the deep learning prediction module and the reinforcement learning optimization module, thereby greatly accelerating the model training and inference process.

[0057] When the system is working, the above modules work together in accordance with the process described in Example 1, forming an automated processing pipeline from raw image input to high-precision change detection result output, which can efficiently and accurately complete large-scale remote sensing change detection tasks.

[0058] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0059] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A remote sensing change detection method based on deep reinforcement learning, characterized in that, include: A deep learning network is used to extract multi-scale deep features and compare differences between paired multi-temporal remote sensing images to generate a preliminary change prediction map and a multi-scale feature pyramid. The preliminary change prediction map represents the probability of changes in the position of each pixel, and the multi-scale feature pyramid contains feature maps with contextual information extracted by different network depths. A reinforcement learning module is introduced, which takes the preliminary change prediction map and the multi-scale feature pyramid as the initial state. Through a decision-making process that includes a state space, an action space and a reward function, the preliminary change prediction map is iteratively optimized in multiple rounds to generate the final change detection result.

2. The remote sensing change detection method based on deep reinforcement learning according to claim 1, characterized in that, The step of generating a preliminary change prediction map using a deep learning network includes: A twin network structure is adopted, wherein the twin network includes an encoder with shared weights, and the encoder extracts multi-scale features of the multi-temporal remote sensing images respectively; Feature difference modeling is performed on the extracted multi-scale features to obtain differential features that reflect image changes; The difference features are input into the decoder, which performs upsampling and feature fusion processing to output the preliminary change prediction map.

3. The remote sensing change detection method based on deep reinforcement learning according to claim 1, characterized in that, The input state of the reinforcement learning module consists of three parts: The deep feature pyramid is extracted by the encoder of the deep learning network and processed by the multi-scale feature fusion module; The segmentation prediction results of the previous iteration, wherein the preliminary change prediction map was used as the segmentation prediction results in the first iteration; The difference map between the previous prediction results and the true labels is used to guide the policy network to focus on the boundary of change and difficult areas.

4. The remote sensing change detection method based on deep reinforcement learning according to claim 1, characterized in that, The reinforcement learning module guides the iterative optimization process through the reward function R; The reward function R is defined as follows: in, and and represent the intersection-union ratio and F1 score after performing the optimization actions, respectively. and Then, these are the cross-union ratio and F1 score before performing the optimization action, CrossEntropyLoss is the cross-entropy loss term, and α, β, and γ are preset weight hyperparameters.

5. The remote sensing change detection method based on deep reinforcement learning according to claim 4, characterized in that, The reward function introduces a misclassification penalty enhancement mechanism, which assigns different weights to misclassified pixels through a location weight map. The penalty weights for pixels in the missed detection area and the false alarm area are greater than those for correctly classified pixels. The underreported area is the area that actually changed but was predicted not to have changed; The false alarm area is the area that is actually unchanged but is predicted to change.

6. The remote sensing change detection method based on deep reinforcement learning according to claim 1, characterized in that, A phased training strategy is adopted, specifically including: The first step is to train the deep learning network independently until the model converges; The second step is to freeze the encoder parameters of the deep learning network and start training the policy network in the reinforcement learning module. The third step is to perform joint fine-tuning on the entire model to improve its overall performance.

7. A remote sensing change detection system based on deep reinforcement learning, characterized in that, include: The data preprocessing module is used to normalize and unify the size of the input multi-temporal remote sensing images; The deep learning prediction module, with a built-in Siamese network structure, is used to extract image features and generate preliminary change prediction maps. The reinforcement learning optimization module is used to iteratively optimize the preliminary change prediction map and output the final change detection result. The storage module is used to store the trained model parameters and remote sensing dataset; The processing unit is used to execute the computational tasks of each module.

8. The remote sensing change detection system based on deep reinforcement learning according to claim 7, characterized in that, The reinforcement learning optimization module includes: State building unit, used to build input state containing deep features, previous prediction results and difference map; The multi-scale feature fusion unit is used to fuse and refine the multi-scale features output by the deep learning prediction module. A policy network is used to output pixel-level classification adjustment actions based on the input state; The reward calculation unit is used to calculate the reward value according to the preset reward function to guide the training of the policy network.

9. The remote sensing change detection system based on deep reinforcement learning according to claim 7, characterized in that, The normalization operation of the data preprocessing module adopts the mean-standard deviation normalization method, and the size unification operation is achieved by interpolation to ensure consistent resolution of multi-temporal remote sensing images.

10. The remote sensing change detection system based on deep reinforcement learning according to claim 7, characterized in that, The decoder of the deep learning prediction module uses a transposed convolutional layer to achieve upsampling and fuses multi-scale features output by the encoder through skip connections to improve the detailed representation capability of the initial change prediction map.

Images