Natural resource normalization auxiliary monitoring optimization method based on reinforcement learning

Abstract

The application discloses a natural resource normalization auxiliary monitoring optimization method based on reinforcement learning, belongs to the technical field of reinforcement learning, and comprises the following steps: performing semantic segmentation and space-time alignment according to multi-source remote sensing time sequence images to generate a semantic segmentation graph set; and performing map difference and grid pre-screening according to the generated semantic segmentation graph set to generate a pre-screening change area set. The application significantly improves the automation level of normalization monitoring, the intelligent degree of decision-making, and the utilization efficiency of computing resources by constructing a closed-loop optimization of deep semantic segmentation and double-level reinforcement learning, realizes full-link adaptive optimization from data preprocessing, potential change area screening to fine collaborative decision-making and dynamic parameter optimization, and effectively enhances the stability and generalization capability of the monitoring system in complex application scenarios.

Description

Technical Field

[0001] This invention belongs to the field of reinforcement learning technology, specifically referring to a method for normalized auxiliary monitoring and optimization of natural resources based on reinforcement learning. Background Technology

[0002] Natural resource surveys and monitoring are fundamental work for understanding the current status and changes of all elements such as mountains, water, forests, fields, lakes, grasslands, sand, minerals, and seas. Their results are an important basis for supporting territorial spatial planning, ecological civilization construction, and economic and social development. With the development of remote sensing technology, routine monitoring using multi-source time-series images such as optical and SAR has become the mainstream method. However, faced with massive and high-dimensional remote sensing data, traditional monitoring methods face severe challenges in terms of efficiency, accuracy, and intelligence.

[0003] However, existing methods for optimizing routine auxiliary monitoring of natural resources still have certain shortcomings. The reward function of existing technologies relies heavily on manual prior settings, which leads to the strategy becoming rigid once deployed and difficult to adapt to dynamically changing monitoring environments. In scenarios where real change events are extremely sparse, sparse reward signals can easily cause exploration exhaustion and learning stagnation in reinforcement learning agents. At the same time, most methods use discrete and coarse parameter adjustment methods, lacking a fine-grained adaptive incremental tuning mechanism in the continuous action space, and cannot utilize feedback data during operation to drive the closed-loop self-evolution of strategies and computing resources. Ultimately, this results in the performance and efficiency of the monitoring system failing to continuously improve with the extension of usage time. Therefore, a reinforcement learning-based optimization method for routine auxiliary monitoring of natural resources is proposed. Summary of the Invention

[0004] The purpose of this invention is to provide a method for normalized auxiliary monitoring and optimization of natural resources based on reinforcement learning, so as to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for normalized assisted monitoring and optimization of natural resources based on reinforcement learning, comprising the following steps:

[0006] S1. Based on multi-source remote sensing time-series images, perform semantic segmentation and spatiotemporal alignment to generate a set of semantic segmentation maps;

[0007] S2. Based on the generated semantic segmentation map set, perform map subtraction and gridding pre-screening to generate a pre-screened change region set;

[0008] S3. Based on the generated pre-screened set of change regions, construct a two-level reinforcement learning agent and a meta-evaluation network;

[0009] S4. Generate a pseudo-value assessment benchmark based on the constructed meta-evaluation network and the pre-screened set of change regions;

[0010] S5. Make collaborative decisions based on pseudo-value assessment benchmarks and a two-level reinforcement learning agent;

[0011] S6. Based on the collaborative decision-making results, perform adaptive incremental parameter tuning based on the continuous action space and generate tuning feedback information.

[0012] S7. Based on the optimization feedback information, perform feedback-driven strategy self-optimization and dynamic resource allocation.

[0013] Preferably, in step S1, the accessed multi-source remote sensing time-series images, including optical satellite images, SAR images, and UAV aerial images, undergo preprocessing such as radiometric correction, atmospheric correction, and cloud detection. Image metadata is extracted and organized into a time-series sequence. A pre-trained deep semantic segmentation model is used to predict the category of each image in each phase. Batch inference is performed using an overlapping block strategy, and the segmentation boundary is optimized by combining conditional random field post-processing, while retaining the category prediction confidence score of each pixel. The segmentation results of multiple phases are spatiotemporally aligned by resampling all temporal images to a unified geospatial coordinate system and pixel resolution to eliminate spatial misalignment caused by sensor differences and imaging angle changes, achieving accurate registration at the sub-pixel level. The semantic segmentation results of each phase of the time-series images, including the category label layer, confidence score layer, and georeferenced information, are organized into a structured semantic segmentation map set in chronological order.

[0014] In step S2, based on the generated semantic segmentation map set, map difference calculation is performed. A rolling window method is used to compare the semantic segmentation maps of adjacent time phases pixel by pixel to identify the pixel locations where the categories change. The frequency and type of change for each pixel are counted to generate an original change detection map containing the change location, change category pair, and change timestamp. The study area is divided into uniform grids, and the grids are initially screened according to a preset threshold to remove stable areas with weak change characteristics. Spatial clustering analysis is performed on the pre-screened grids. DBSCAN or region growing is used to merge adjacent candidate grids into continuous change area patches. At the same time, the geometric and semantic features of each patch are calculated. The reliability score of each patch is evaluated by combining a preset confidence propagation mechanism. Finally, a pre-screened change area set containing spatial location, change characteristics, and confidence score is generated.

[0015] Preferably, in step S3, a two-level reinforcement learning agent architecture is constructed based on the generated pre-screened set of change regions, including a high-level agent and a low-level agent.

[0016] The high-level intelligent agent acts as a policy selector, dynamically selecting the target region with the most analytical value from candidate changing regions. The action space is defined as the region selection decision, and the reward function comprehensively considers the region information gain and analysis cost.

[0017] The low-level agent acts as a fine-grained analyzer, performing pixel-level in-depth analysis on each change region selected by the high-level agent. Its state space contains multi-temporal semantic segmentation results, texture features, and contextual information within the region. The action space is defined as pixel-level annotation correction operations. The reward function is designed based on the IoU lift with the reference annotation and the boundary smoothness.

[0018] Simultaneously, a meta-evaluation network is constructed, consisting of a change region quality evaluation module, an agent decision-making effect evaluation module, and an adaptive parameter adjustment module. The quality evaluation module uses a multilayer perceptron to quantify the authenticity, saliency, and analysis difficulty of the change region. The decision-making effect evaluation module calculates the agent's decision accuracy and efficiency indicators based on historical decision records and feedback information. The adaptive parameter adjustment module dynamically adjusts the exploration rate of the high-level agent, the learning rate of the low-level agent, and the weight coefficients of the reward function according to the evaluation results.

[0019] Preferably, in step S4, the pre-screened set of change regions is subjected to multi-dimensional reasoning based on the constructed meta-evaluation network, extracting the spatial geometric features, semantic features, and temporal evolution features of each candidate region. The four parallel branches of the input meta-evaluation network output detection quality scores respectively. Spatiotemporal consistency score Uncertainty score and abnormal probability The presupposed adaptive pseudo-value assessment is:

[0020] ,

[0021] In the formula, This represents the basic pseudo-value score, with values ​​ranging from [0, 1]. Indicates the quality score of the test. U represents the spatiotemporal consistency score, and U represents the uncertainty score. Indicates the probability of an anomaly. This represents the weighting index for testing quality. Indicates the spatiotemporal consistency weight index. Indicates the reliability weighting index. Indicates the abnormality suppression weight index

[0022] Preferably, in step S4, based on the basic pseudo-value score, a spatiotemporal context attention mechanism is introduced to calculate the similarity weighted average between the candidate region and its K spatial neighbors, as follows:

[0023] ,

[0024] In the formula, This represents the final pseudo-value score. Let K represent the base value of the i-th neighborhood, and K represent the number of neighborhoods. Represents the neighborhood weight. for , Indicates distance, Indicates the attenuation coefficient. Indicates the proportion of neighborhood fusion. ;

[0025] Based on the final pseudo-value score Dynamically generate pseudo-label confidence weights for reinforcement learning training, as follows:

[0026] ,

[0027] In the formula, This represents the confidence weight of the pseudo-label, with a value range of [0, 1]. It represents the confidence sensitivity parameter; and generates a pseudo-value assessment benchmark that includes region ID, spatial boundary, final pseudo-value score, pseudo-label type, and pseudo-label confidence weight.

[0028] Preferably, in step S5, the high-level intelligent agent reads the generated pseudo-value evaluation benchmark, regards each candidate region and its value score and confidence weight as the global environment state, and outputs macro-decision actions based on the current state, and sends sub-objectives to the low-level intelligent agent;

[0029] The lower-level agent receives sub-targets from the higher-level agent and uses the original dual-temporal image blocks of the region as the local observation environment. Under the constraints of the sub-targets, the lower-level agent performs high-frequency fine-tuning operations in the local region.

[0030] When the lower-level agent completes the task in the current area or reaches the maximum number of attempts and actively reports a termination signal, the higher-level agent is reactivated and selects the next target area; the higher-level agent remains silent during the execution of the lower-level agent.

[0031] Preferably, in step S6, the results of collaborative decision-making are feature-fused and mapped into a continuous state vector. The action space is defined as a continuous parameter increment vector. Simultaneously, an exploration mechanism is introduced, with a pre-defined constrained incremental action generation formula, implemented as follows:

[0032] ,

[0033] In the formula, and These represent the system control parameter vector at the current moment and the parameter vector at the next moment after optimization, respectively. The output of the policy network is represented by the current state. Predict a specific direction for basic optimization. This represents random noise that follows a Gaussian distribution, used for creative exploration in a continuous action space to avoid the strategy getting trapped in local optima. This indicates the intensity of the exploration. This represents the hyperbolic tangent activation function. The basic step size hyperparameter has a range of values ​​(0, +∞), i.e., any positive real number. Indicates the exploration coefficient. This represents a collaborative decision-making mask.

[0034] Preferably, in step S6, the actual value of the incremental action is accurately evaluated and used to guide the update of the policy network. This is achieved through value evaluation incorporating the idea of ​​generalized advantage estimation.

[0035] ,

[0036] In the formula, This represents the estimated advantage function value for the parameter increment action at the current moment. This represents the instant reward function. and This represents the Critic value network's long-term value assessment of the current state and the next state. This represents the discount factor, with a value range of [0, 1], used to balance the weight of immediate rewards and future long-term returns. This represents the generalized advantage estimation decay factor, with a value range of [0, 1]. The advantage value... Parameter convergence status and final optimized parameters Encapsulated as optimization feedback information.

[0037] Preferably, in step S7, real-time optimization feedback information is collected, including parameter convergence status, advantage evaluation value, and actual performance indicators after task execution. For optimization results that are unsatisfactory or fail to meet expectations, in-depth attribution analysis is performed. The feedback signal after attribution analysis is used as a reward or penalty signal in reinforcement learning and propagated back to the Critic value network. Based on the optimization difficulty in the feedback, the complexity of the current task is evaluated in real time. If the system detects that the changing characteristics of the current region are extremely complex, requiring larger-scale exploration or finer boundary fine-tuning, it is determined to be a high-load task. Elastic scaling of computing resources: Based on the evaluation results, the computing resource quotas allocated to high- and low-level agents are dynamically adjusted. The updated strategy and new resource allocation scheme are executed, and their actual performance is closely monitored in the next round. By comparing key indicators before and after optimization, the effectiveness of self-optimization and resource reallocation is verified.

[0038] Compared with the prior art, the beneficial effects of the present invention are:

[0039] 1. To address the problems of low processing efficiency, high reliance on manual intervention, and difficulty in adapting to complex surface dynamics when dealing with massive amounts of multi-source remote sensing data in traditional natural resource monitoring methods, and to overcome the limitations of single models in terms of spatiotemporal alignment accuracy and robustness of change detection, this invention significantly improves the automation level, decision-making intelligence, and computing resource utilization efficiency of routine monitoring by constructing a closed-loop optimization of deep semantic segmentation and two-level reinforcement learning. It achieves end-to-end adaptive optimization from data preprocessing and potential change area screening to refined collaborative decision-making and dynamic parameter tuning, effectively enhancing the stability and generalization ability of the monitoring system in complex application scenarios.

[0040] 2. This invention completely eliminates the dependence of existing methods on manual prior reward functions by constructing a pseudo-value evaluation benchmark based on multi-dimensional feature fusion and spatiotemporal context attention mechanism; the meta-evaluation network adaptively extracts the spatial geometric features, semantic purity and continuity index of candidate regions without relying on external labels, and calculates the final pseudo-value score using a neighborhood fusion strategy based on distance decay; the reward signal is transformed from an externally given signal to an endogenous constraint based on the region's own physical attributes and neighborhood consistency, and the system can automatically identify high-value regions and generate confidence weights;

[0041] 3. This invention solves the problem of reinforcement learning exploration exhaustion caused by the extreme sparseness of real change events through a two-level collaborative architecture of high-level and low-level agents. The high-level agent acts as a policy selector, macroscopically locking high-value regions based on pseudo-value benchmarks and issuing sub-targets. The low-level agent acts as a fine-grained analyzer, performing high-frequency pixel-level boundary correction and semantic confirmation under the constraints of sub-targets. The two achieve decoupling and collaboration from macro-region selection to micro-pixel operations through a strict hierarchical interaction protocol and a two-way value feedback mechanism.

[0042] 4. This invention solves the problems of strategy deployment fixation and low resource utilization in existing systems by using adaptive incremental parameter tuning and closed-loop feedback mechanisms in a continuous action space. The strategy network outputs continuous parameter increments with exploratory noise, and performs micro-step updates with smooth gradients to avoid oscillations in discrete actions. More importantly, the system establishes a load-aware resource elastic scaling mechanism, dynamically adjusting the number of inference cores and memory bandwidth quotas according to the task complexity in the tuning feedback. This ensures high accuracy and high throughput when dealing with complex geological changes, and enables monitoring efficiency to continuously increase with the extension of usage time. Attached Figure Description

[0043] Figure 1 This is the operational flow of the reinforcement learning-based normalized assisted monitoring and optimization method for natural resources according to the present invention. Figure One ;

[0044] Figure 2 This is the operational flow of the reinforcement learning-based normalized assisted monitoring and optimization method for natural resources according to the present invention. Figure Two ;

[0045] Figure 3 This is the operational flow of the reinforcement learning-based normalized assisted monitoring and optimization method for natural resources according to the present invention. Figure Three ;

[0046] Figure 4 This is the operational flow of the reinforcement learning-based normalized assisted monitoring and optimization method for natural resources according to the present invention. Figure Four ;

[0047] Figure 5 This is the operational flow of the reinforcement learning-based normalized assisted monitoring and optimization method for natural resources according to the present invention. Figure Five . Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] Example

[0050] Please see Figures 1-5 As shown, the present invention provides a technical solution comprising the following steps:

[0051] S1. Based on multi-source remote sensing time-series images, perform semantic segmentation and spatiotemporal alignment to generate a set of semantic segmentation maps;

[0052] S2. Based on the generated semantic segmentation map set, perform map subtraction and gridding pre-screening to generate a pre-screened change region set;

[0053] S3. Based on the generated pre-screened set of change regions, construct a two-level reinforcement learning agent and a meta-evaluation network;

[0054] S4. Generate a pseudo-value assessment benchmark based on the constructed meta-evaluation network and the pre-screened set of change regions;

[0055] S5. Make collaborative decisions based on pseudo-value assessment benchmarks and a two-level reinforcement learning agent;

[0056] S6. Based on the collaborative decision-making results, perform adaptive incremental parameter tuning based on the continuous action space and generate tuning feedback information.

[0057] S7. Based on the optimization feedback information, perform feedback-driven strategy self-optimization and dynamic resource allocation.

[0058] In this embodiment, in step S1, the accessed multi-source remote sensing time-series images are preprocessed with radiometric correction, atmospheric correction, and cloud detection, including optical satellite images, SAR images, UAV aerial images, etc., and image metadata is extracted and organized into a time-series sequence in chronological order; each image is predicted pixel by pixel using a pre-trained deep semantic segmentation model, using an encoder-decoder architecture such as U-Net or DeepLabv3+, and the backbone network can be ResNet or SwinTransformer;

[0059] Batch inference is performed using an overlapping block strategy, and the segmentation boundary is optimized by post-processing with a conditional random field while retaining the category prediction confidence score of each pixel. Spatiotemporal alignment is performed on the segmentation results of multiple periods, and corresponding control points are extracted using a feature matching algorithm to calculate geometric transformation parameters, such as affine transformation or polynomial transformation coefficients.

[0060] All temporal images are resampled to a unified geospatial coordinate system and pixel resolution to eliminate spatial misalignment caused by sensor differences and imaging angle changes, achieving accurate registration at the sub-pixel level. The semantic segmentation results of each temporal image, including the category label layer, confidence score layer, and georeferenced information, are organized into a structured semantic segmentation map set in chronological order.

[0061] In step S2, based on the generated semantic segmentation map set, map difference calculation is performed. A rolling window method is used to compare the semantic segmentation maps of adjacent time phases pixel by pixel to identify the pixel positions where the category changes. The frequency and type of change of each pixel are counted to generate an original change detection map containing the change position, change category pair and change timestamp.

[0062] Specifically, to reduce computational complexity and focus on potential change areas, the study area is divided into uniform grids, such as 100×100 meters or 512×512 pixels. Change statistical analysis is performed on each grid cell to calculate the change cell density, change category diversity index, and change spatial clustering within the grid.

[0063] Specifically, the grid is initially screened based on a preset threshold to remove stable regions with weak variation characteristics. Spatial clustering analysis is then performed on the pre-screened grids, and adjacent candidate grids are merged into continuous variation region patches using DBSCAN or region growing algorithms. At the same time, the geometric features (area, perimeter, shape index) and semantic features (dominant variation type, variation intensity) of each patch are calculated. The reliability score of each patch is evaluated in conjunction with a preset confidence propagation mechanism, and finally, a set of pre-screened variation regions containing spatial location, variation characteristics, and confidence scores is generated.

[0064] In this embodiment, in step S3, a two-level reinforcement learning agent architecture is constructed based on the generated pre-screened change region set, including a high-level agent and a low-level agent.

[0065] The high-level agent acts as a policy selector, dynamically selecting the target region with the most analytical value from the candidate changing regions. Its state space includes the geometric features, semantic features and confidence scores of the region, and the action space is defined as the region selection decision. The reward function comprehensively considers the region information gain and analysis cost.

[0066] The low-level agent acts as a fine-grained analyzer, performing pixel-level in-depth analysis on each change region selected by the high-level agent. Its state space contains multi-temporal semantic segmentation results, texture features, and contextual information within the region, while its action space is defined as pixel-level annotation and correction operations, such as category adjustment and boundary optimization.

[0067] The reward function is designed based on the IoU lift and boundary smoothness with reference labels; the two-layer agents collaborate through a hierarchical decision-making mechanism. The policy network of the high-level agent is trained using the PPO or SAC algorithm, while the policy network of the low-level agent is trained using the DQN or A2C algorithm. They share some feature extraction layers to achieve knowledge transfer.

[0068] Simultaneously, a meta-evaluation network is constructed, which consists of a change region quality evaluation module, an agent decision-making effect evaluation module, and an adaptive parameter adjustment module. The quality evaluation module uses a multilayer perceptron to quantify the authenticity, saliency, and analysis difficulty of the change region. The decision-making effect evaluation module calculates the agent's decision accuracy and efficiency indicators based on historical decision records and feedback information. The adaptive parameter adjustment module dynamically adjusts the exploration rate of the high-level agent, the learning rate of the low-level agent, and the weight coefficient of the reward function according to the evaluation results, forming a closed-loop optimization mechanism.

[0069] The entire system is iteratively optimized through interactive training. The agent continuously accumulates experience during the analysis process, and the meta-evaluation network continuously monitors the system performance and provides tuning suggestions, ultimately forming a collaborative framework of a two-level reinforcement learning agent and a meta-evaluation network with adaptive learning capabilities.

[0070] In this embodiment, in step S4, multi-dimensional reasoning is performed on the pre-screened set of change regions based on the constructed meta-evaluation network to extract the spatial geometric features (area A, perimeter P, shape index) of each candidate region. The input features (class purity CP, diversity entropy H) and temporal evolution features (rate of change R, continuity index CI) are used to input the four parallel branches of the meta-evaluation network, which output detection quality scores respectively. Spatiotemporal consistency score Uncertainty score and abnormal probability The presupposed adaptive pseudo-value assessment is:

[0071] ,

[0072] In the formula, This represents the basic pseudo-value score, with values ​​ranging from [0, 1]. Indicates the quality score of the test. U represents the spatiotemporal consistency score, and U represents the uncertainty score. Indicates the probability of an anomaly. This represents the weighting index for testing quality. Indicates the spatiotemporal consistency weight index. Indicates the reliability weighting index. Indicates the abnormal suppression weight index;

[0073] Based on information fusion theory, a multiplicative fusion strategy is adopted to integrate positive indicators. Direct multiplication, negative index Converted to inhibitory factor , Exponential parameters Controlling the weights of each dimension, when and hour Reaching the theoretical maximum value when any indicator is 0 It meets the [0, 1] normalization requirement.

[0074] In this embodiment, in step S4, based on the basic pseudo-value score, a spatiotemporal context attention mechanism is introduced to calculate the similarity weighted average between the candidate region and its K spatial neighbors, which is implemented as follows:

[0075] ,

[0076] In the formula, This represents the final pseudo-value score. Let K represent the base value of the i-th neighborhood, and K represent the number of neighborhoods. Represents the neighborhood weight. for , Indicates distance, Indicates the attenuation coefficient. Indicates the proportion of neighborhood fusion. ;

[0077] Based on Markov random field theory, the evaluation results of the neighborhood region should influence the final value of the central region, with weights... Based on spatial distance calculate( (distance attenuation coefficient) By controlling the intensity of neighborhood influence, this formula ensures that spatially adjacent high-value regions reinforce each other, thereby improving the spatial consistency of the pseudo-benchmark.

[0078] Based on the final pseudo-value score Dynamically generate pseudo-label confidence weights for reinforcement learning training, as follows:

[0079] ,

[0080] In the formula, This represents the confidence weight of the pseudo-label, with a value range of [0, 1]. It represents the confidence sensitivity parameter; and generates a pseudo-value assessment benchmark that includes region ID, spatial boundary, final pseudo-value score, pseudo-label type, and pseudo-label confidence weight.

[0081] In this embodiment, in step S5, the high-level intelligent agent reads the generated pseudo-value evaluation benchmark and regards each candidate region and its value score and confidence weight as the global environment state. Instead of directly processing image pixels, it evaluates from a macro perspective which regions are most worthy of priority processing, such as the regions with the highest value or the greatest uncertainty.

[0082] Based on the current state, the higher-level agent outputs macro-level decision actions and issues sub-goals to the lower-level agents. The actions are not specific pixel annotations, but rather a sub-goal or instruction is issued to the lower-level agents; for example, locking in a high-value candidate region and specifying the annotation quality standard or change type tendency that the region needs to achieve.

[0083] The high-level intelligent agent makes decisions at a low frequency. Once a sub-goal is issued, the instruction will remain stable for a long period of time, continuously guiding the work of the low-level intelligent agent until the labeling task of the current area is judged to be completed or failed. Only then will the high-level intelligent agent intervene to select the next area.

[0084] The lower-level agent receives sub-targets from the higher-level agent, such as the coordinate range of the target area and the expected change attributes. It uses the original dual-temporal image patch of the area as the local observation environment. Under the constraints of the sub-targets, the lower-level agent performs high-frequency fine-tuning operations in the local area.

[0085] Actions include pixel-level boundary fine-tuning, morphological correction, and confirmation of local semantic categories. Through continuous trial and error and exploration, fine-grained change masks that meet the requirements of higher levels are gradually generated. The decision frequency of the lower-level agent is extremely high, focusing on optimizing the current local annotation effect to the best in a very short time scale through multiple actions, so as to meet the sub-goals set by the higher-level agent as much as possible.

[0086] A strict hierarchical interaction protocol is established. When a lower-level agent completes the task in the current area or reaches the maximum number of attempts and actively reports a termination signal, the higher-level agent is reactivated and selects the next target area. During the execution of the lower-level agent, the higher-level agent remains silent to avoid decision oscillations caused by frequent intervention.

[0087] Two-way value feedback:

[0088] Bottom-up feedback: After the lower-level agent completes the action, it will feed back the actual quality of the annotation results, such as the actual fit and the smoothness of the boundary, to the higher-level agent.

[0089] Top-down correction: Based on the feedback results and combined with the pseudo-value benchmark in S4, the high-level agent dynamically updates the value assessment of the remaining unprocessed areas. If the success rate of the low-level agents in a certain type of area is consistently low, the high-level agent will adjust the priority of that type of area or change the sub-objective strategy issued in subsequent decisions.

[0090] A pre-defined differentiated reward mechanism is used: high-level agents receive delayed rewards (sparse rewards) based on the overall confidence of the final generated pseudo-labels; while low-level agents receive immediate local rewards (dense rewards) based on whether each of their actions makes the labeling results closer to the sub-target.

[0091] Progressive learning from easy to difficult: In the early stages of training, the higher-level agent is forced to prioritize issuing instructions to regions with extremely high confidence (simple samples) in the S4 benchmark, helping the lower-level agent quickly master basic annotation actions. As training progresses, the selection restrictions are gradually relaxed, guiding the agent to actively explore low-confidence (difficult samples) regions, achieving robust transfer of decision-making capabilities from simple to complex.

[0092] In this embodiment, in step S6, the results of collaborative decision-making are fused with features such as the sub-target instructions of the high-level agent, the execution feedback of the low-level agent, the confidence distribution of the current pseudo-label, and the historical tuning error, and mapped to a continuous state vector. This vector comprehensively reflects the system's operational deviations and optimization potential under the current parameter configuration.

[0093] Specifically, the action space is defined as a continuous parameter increment vector. The action is no longer about directly setting parameter values, but rather outputting a fine-tuning amount, enabling the system to adjust the current parameters. Based on this, refined adaptive correction is performed. To prevent excessive parameter adjustments in the continuous action space from causing system instability, an exploration mechanism is introduced, and a pre-defined constrained incremental action generation formula is implemented as follows:

[0094] ,

[0095] In the formula, and These represent the system control parameter vector at the current moment and the parameter vector at the next moment after optimization, respectively. The output of the policy network is represented by the current state. Predict a specific direction for basic optimization. This represents random noise that follows a Gaussian distribution, used for creative exploration in a continuous action space to avoid the strategy getting trapped in local optima. This indicates the intensity of the exploration. Let represent the hyperbolic tangent activation function, which forces the weighted sum of the basic tuning direction and random noise to compress it into the interval (−1, 1). The basic step size hyperparameter has a range of values ​​(0, +∞), i.e., any positive real number. Indicates the exploration coefficient. This represents the collaborative decision mask, which is generated by the high-level agent of S5. It is a 0-1 vector with the same dimension as the parameter vector and is used to dynamically mask the parameter dimensions that do not need to be adjusted or are sensitive, so as to achieve targeted local parameter tuning.

[0096] In this embodiment, in step S6, the actual value of the incremental action is accurately evaluated and the policy network update is guided. This is achieved through value evaluation incorporating the idea of ​​generalized advantage estimation.

[0097] ,

[0098] In the formula, This represents the estimated advantage function value for the parameter increment action at the current moment. This represents the instant reward function. and This represents the Critic value network's long-term value assessment of the current state and the next state. This represents the discount factor, with a value range of [0, 1], used to balance the weight of immediate rewards and future long-term returns. This represents the generalized advantage estimation decay factor, with a value range of [0, 1]. The advantage value... Parameter convergence status and final optimized parameters Encapsulated as optimization feedback information.

[0099] In this embodiment, in step S7, optimization feedback information is collected in real time, including parameter convergence status, advantage evaluation value, actual performance indicators after task execution, etc. The multi-source heterogeneous raw data is cleaned, denoised and standardized to transform it into a structured feedback signal. For optimization results that are not performing well or do not meet expectations in the feedback, in-depth attribution analysis is performed.

[0100] Specifically, the problem is precisely located at a specific stage, such as whether there is a deviation in the sub-goal setting of the high-level agent, the wrong direction of the incremental action exploration of the low-level agent, or whether the currently allocated computing resources are insufficient to support complex parameter searches such as inference computing power and memory bandwidth. The feedback signal after attribution analysis is used as a reward or punishment signal in reinforcement learning and backpropagated to the Critic value network. If an incremental optimization brings a significant performance improvement (high advantage value), the probability of the policy network outputting such an action in that state is strengthened; otherwise, it is suppressed.

[0101] A long short-term memory mechanism is established to store high-quality state-action-feedback trajectories in the experience replay pool. During training or online inference, the agent retrieves historical successful experiences to assist in current decision-making. For patterns that repeatedly fail, the system will automatically adjust the exploration coefficient or modify the goal setting rules of the higher-level agent to achieve adaptive optimization of the strategy.

[0102] Based on the feedback on the difficulty of optimization, such as the number of steps required for parameter convergence and the fluctuation range of the advantage value, the complexity of the current task is evaluated in real time. If the system detects that the changing characteristics of the current region are extremely complex and require larger-scale exploration or finer boundary fine-tuning, it is determined to be a high-load task.

[0103] Elastic scaling of computing resources: Based on the evaluation results, the computing resource quotas allocated to high- and low-level agents are dynamically adjusted. For example, for high-difficulty tuning tasks, the number of GPU inference cores is automatically increased, the sampling batch of the experience replay pool is expanded, or the action execution frequency of low-level agents is increased. For simple tasks, resources are appropriately reduced and released to other parallel processing threads. The updated strategy and new resource allocation scheme are implemented, and their actual performance is closely monitored in the next round. By comparing key indicators before and after optimization, such as the accuracy of pseudo-label generation and the convergence speed of parameter tuning, the effectiveness of self-optimization and resource reallocation is verified.

[0104] Working Principle: Preprocessing operations such as radiometric correction, atmospheric correction, and cloud detection are performed on multi-source remote sensing time-series images to unify image data from different sources to a comparable standard. Then, a pre-trained deep semantic segmentation model is used to predict the category of each image in each phase, employing an encoder-decoder architecture to extract multi-level features and generate pixel-level classification results. Next, spatiotemporal alignment processing is performed on the segmentation results from multiple phases. Corresponding control points are extracted using a feature matching algorithm, and geometric transformation parameters are calculated. All temporal images are resampled to a unified geospatial coordinate system to eliminate spatial misalignment caused by sensor differences and imaging angle variations. Finally, the semantic segmentation results of each phase of the time-series images are organized into a structured semantic segmentation map set in chronological order. Map difference calculation is performed based on the generated semantic segmentation map set. A rolling window approach is used to compare the semantic segmentation maps of adjacent time phases pixel by pixel to identify the locations of pixels where the category changes and to count the frequency and type of changes. To reduce computational complexity and focus on potential change areas, the study area is divided into uniform grids and statistical analysis of changes is performed on each grid cell to calculate indicators such as the change pixel density, change category diversity index, and spatial clustering. The grids are initially screened based on a preset threshold to remove stable areas with weak change characteristics. Then, spatial clustering analysis is performed on the pre-screened grids to merge adjacent candidate grids into continuous change area patches, and the geometric and semantic features of each patch are calculated. The reliability score is evaluated using a confidence propagation mechanism, and finally, a set of pre-screened change areas containing spatial location change features and confidence scores is generated.

[0105] A two-tiered reinforcement learning agent architecture is constructed based on the generated pre-selected set of change regions, including a high-level agent and a low-level agent. The high-level agent acts as a policy selector, dynamically selecting the most valuable target region from the candidate change regions. Its state space includes the region's geometric and semantic features and confidence scores. The low-level agent acts as a refined analyzer, performing pixel-level in-depth analysis on each change region selected by the high-level agent. Its state space includes the texture features of the multi-temporal semantic segmentation results within the region and contextual information. The two agents collaborate through a hierarchical decision-making mechanism. The policy network of the high-level agent is trained using proximal policy optimization or the soft actor critic algorithm, while the policy network of the low-level agent is trained using a deep network or the dominant actor critic algorithm. They share some feature extraction layers to achieve knowledge transfer. A meta-evaluation network is also constructed, consisting of a change region quality evaluation module, an agent decision performance evaluation module, and an adaptive parameter adjustment module. Based on this construction... The meta-evaluation network performs multi-dimensional reasoning on a pre-screened set of change regions, extracting spatial geometric features, semantic features, and temporal evolution features of each candidate region. The input to the four parallel branches of the meta-evaluation network outputs detection quality score, spatiotemporal consistency score, uncertainty score, and anomaly probability, respectively. A multiplicative fusion strategy is adopted to directly multiply positive indicators and convert negative indicators into suppression factors. The exponential parameter controls the weights of each dimension to generate a basic pseudo-value score. Then, based on the basic pseudo-value score, a spatiotemporal context attention mechanism is introduced to calculate the similarity weight between the candidate region and its spatial neighborhood. Based on Markov random field theory, the evaluation results of the neighborhood regions affect the final value of the central region. The weights are calculated based on spatial distance to ensure that spatially adjacent high-value regions mutually reinforce each other. Finally, pseudo-label confidence weights are dynamically generated based on the final pseudo-value score for reinforcement learning training, generating a pseudo-value evaluation benchmark that includes the region identifier, spatial boundary, final pseudo-value score, pseudo-label type, and pseudo-label confidence weight.

[0106] The high-level agent reads the generated pseudo-value assessment benchmark, treating each candidate region and its value score confidence weight as the global environment state, and assesses from a macro perspective which regions are most worthy of priority processing. Based on the current state, the high-level agent outputs macro-level decision actions, issuing sub-objectives or instructions to the low-level agent, such as locking a high-value candidate region and specifying the annotation quality standard that the region needs to achieve. The high-level agent makes decisions at a low frequency, and once a sub-objective is issued, the instruction remains stable for a relatively long period of time. The low-level agent receives the sub-objectives issued by the high-level agent, uses the original dual-temporal image patch of the region as the local observation environment, and performs high-frequency fine-tuning operations under the constraints of the sub-objectives, including pixel-level boundary fine-tuning and morphological repair. The system confirms the local semantic category; establishes a strict hierarchical interaction protocol, whereby the higher-level agent reactivates and selects the next target region when the lower-level agent completes the task in the current region or reaches the maximum number of attempts; simultaneously establishes a two-way value feedback mechanism, where the lower-level agent feeds back the actual quality of the annotation results to the higher-level agent, and the higher-level agent dynamically updates the value assessment of the remaining unprocessed regions based on the feedback results; pre-sets a differentiated reward mechanism, where the higher-level agent receives a delayed reward based on the overall confidence of the final generated pseudo-labels, and the lower-level agent receives an immediate local reward based on whether each of its actions makes the annotation result closer to the sub-target; and performs feature fusion on the results of collaborative decision-making, including the sub-target instructions from the higher-level agent and the execution of those instructions by the lower-level agent. The system feeds back the confidence distribution of the current pseudo-labels and historical tuning errors, mapping them to a continuous state vector. This vector comprehensively reflects the system's operational deviations and optimization potential under the current parameter configuration. The action space is defined as a continuous parameter increment vector. Actions no longer directly set parameter values ​​but output a fine-tuning amount, enabling the system to perform refined adaptive corrections based on the current parameters. To prevent excessive parameter adjustments in the continuous action space from causing system instability, an exploration mechanism is introduced. A constrained incremental action generation formula is designed, and the weighted sum of the basic tuning direction and random noise is forcibly compressed to the range of negative one to one using a hyperbolic tangent activation function. A collaborative decision mask is introduced to achieve targeted local parameter tuning. Then, fine-tuning is performed. The system accurately assesses the actual value of incremental actions by calculating the estimated advantage function using a value assessment method incorporating the idea of ​​generalized advantage estimation. It encapsulates the convergence status of the advantage parameters and the final optimized parameters as optimization feedback information. Real-time acquisition of optimization feedback information, including parameter convergence status, advantage assessment values, and actual performance indicators after task execution, is performed. The system cleans, denoises, and standardizes the multi-source heterogeneous raw data, transforming it into structured feedback signals. In-depth attribution analysis is conducted on optimization results that are unsatisfactory or fail to meet expectations, pinpointing the problem to specific stages, such as whether there are deviations in the sub-objective settings of higher-level agents, errors in the incremental action exploration direction of lower-level agents, or insufficient allocated computing resources to support complex parameter searches.The feedback signals from attribution analysis are used as reward or penalty signals in reinforcement learning, propagated back to the value network to reinforce advantageous actions and suppress disadvantageous strategies. A long short-term memory mechanism is established to store high-quality state-action feedback trajectories in an experience replay pool, retrieving historical successful experiences during training or online inference to assist current decisions. For patterns that repeatedly fail, the exploration coefficient is automatically adjusted or the goal-setting rules of higher-level agents are modified. The complexity of the current task is evaluated in real time based on the optimization difficulty of the feedback; if the system detects extremely complex changes in the current region, it is determined to be a high-load task. Based on the evaluation results, the computational resource quotas allocated to high- and low-level agents are dynamically adjusted, automatically increasing computational resources for high-difficulty optimization tasks and appropriately reducing resources for simple tasks. The updated strategy and new resource allocation scheme are executed, and their actual performance is closely monitored in the next round. The effectiveness of self-optimization and resource reallocation is verified by comparing key indicators before and after optimization.

[0107] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their likenesses.

[0108] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. A method for routine auxiliary monitoring and optimization of natural resources based on reinforcement learning, characterized in that, Includes the following steps: S1. Based on multi-source remote sensing time-series images, perform semantic segmentation and spatiotemporal alignment to generate a set of semantic segmentation maps; S2. Based on the generated semantic segmentation map set, perform map subtraction and gridding pre-screening to generate a pre-screened change region set; S3. Based on the generated pre-screened set of change regions, construct a two-level reinforcement learning agent and a meta-evaluation network; S4. Generate a pseudo-value assessment benchmark based on the constructed meta-evaluation network and the pre-screened set of change regions; S5. Make collaborative decisions based on pseudo-value assessment benchmarks and a two-level reinforcement learning agent; S6. Based on the collaborative decision-making results, perform adaptive incremental parameter tuning based on the continuous action space and generate tuning feedback information. S7. Based on the optimization feedback information, perform feedback-driven strategy self-optimization and dynamic resource allocation.

2. The method for normalized auxiliary monitoring and optimization of natural resources based on reinforcement learning according to claim 1, characterized in that: In step S1, the accessed multi-source remote sensing time-series images are preprocessed to extract image metadata and organize it into a time-series sequence in chronological order. A pre-trained deep semantic segmentation model is used to predict the category of each image in each phase. Batch inference is performed using an overlapping block strategy, and the segmentation boundary is optimized by post-processing with a conditional random field, while retaining the category prediction confidence score of each pixel. The segmentation results of multiple phases are spatiotemporally aligned by resampling all temporal images to a unified geospatial coordinate system and pixel resolution to eliminate spatial misalignment caused by sensor differences and imaging angle changes, achieving accurate registration at the sub-pixel level. The semantic segmentation results of each phase of time-series images are organized into a structured semantic segmentation map set in chronological order.

3. The method for normalized auxiliary monitoring and optimization of natural resources based on reinforcement learning according to claim 1, characterized in that: In step S2, based on the generated semantic segmentation map set, a rolling window is used to compare the semantic segmentation maps of adjacent time phases pixel by pixel to identify the pixel positions where the categories change, and to count the change frequency and change type of each pixel. An original change detection map containing the change position, change category pair and change timestamp is generated. The study area is divided into a uniform grid, and the grid is initially screened according to a preset threshold to remove stable areas with weak change characteristics. Spatial clustering analysis is performed on the pre-screened grids to merge adjacent candidate grids into continuous variation region patches. At the same time, the geometric and semantic features of each patch are calculated, and finally a set of pre-screened variation regions is generated.

4. The method for normalized assisted monitoring and optimization of natural resources based on reinforcement learning according to claim 1, characterized in that: In S3, a two-level reinforcement learning agent architecture is constructed based on the generated pre-screened change region set, including a high-level agent and a low-level agent. The high-level intelligent agent acts as a policy selector, dynamically selecting the target region with the most analytical value from candidate changing regions. The action space is defined as the region selection decision, and the reward function comprehensively considers the region information gain and analysis cost. The low-level agent acts as a fine-grained analyzer, performing pixel-level in-depth analysis of each region of change selected by the high-level agent; Simultaneously, a meta-evaluation network is constructed, consisting of a change region quality evaluation module, an agent decision-making effect evaluation module, and an adaptive parameter adjustment module. The quality evaluation module uses a multilayer perceptron to quantify the authenticity, saliency, and analysis difficulty of the change region. The decision-making effect evaluation module calculates the agent's decision accuracy and efficiency indicators based on historical decision records and feedback information. The adaptive parameter adjustment module dynamically adjusts the exploration rate of the high-level agent, the learning rate of the low-level agent, and the weight coefficients of the reward function according to the evaluation results.

5. The method for normalized assisted monitoring and optimization of natural resources based on reinforcement learning according to claim 1, characterized in that: In step S4, the pre-screened set of change regions is subjected to multi-dimensional reasoning based on the constructed meta-evaluation network. The spatial geometric features, semantic features, and temporal evolution features of each candidate region are extracted, and the four parallel branches of the meta-evaluation network output detection quality scores respectively. Spatiotemporal consistency score Uncertainty score and abnormal probability The presupposed adaptive pseudo-value assessment is: ， In the formula, Indicates the basic pseudo-value score. Indicates the quality score of the test. U represents the spatiotemporal consistency score, and U represents the uncertainty score. Indicates the probability of an anomaly. This represents the weighting index for testing quality. Indicates the spatiotemporal consistency weight index. Indicates the reliability weighting index. This represents the abnormality suppression weight index.

6. The method for normalized auxiliary monitoring and optimization of natural resources based on reinforcement learning according to claim 1, characterized in that: In step S4, based on the basic pseudo-value score, a spatiotemporal context attention mechanism is introduced to calculate the similarity weighted average between the candidate region and its K spatial neighbors, as follows: ， In the formula, This represents the final pseudo-value score. Let K represent the base value of the i-th neighborhood, and K represent the number of neighborhoods. Represents the neighborhood weight. for , Indicates distance, Indicates the attenuation coefficient. Indicates the proportion of neighborhood fusion; Based on the final pseudo-value score Dynamically generate pseudo-label confidence weights to generate a pseudo-value assessment benchmark that includes region ID, spatial boundary, final pseudo-value score, pseudo-label type, and pseudo-label confidence weights.

7. The method for normalized assisted monitoring and optimization of natural resources based on reinforcement learning according to claim 1, characterized in that: In S5, the high-level intelligent agent reads the generated pseudo-value evaluation benchmark, regards each candidate region and its value score and confidence weight as the global environment state, and outputs macro-decision actions based on the current state, and sends sub-objectives to the low-level intelligent agent. The lower-level agent receives sub-targets from the higher-level agent and uses the original dual-temporal image blocks of the region as the local observation environment. Under the constraints of the sub-targets, the lower-level agent performs high-frequency fine-tuning operations in the local region. When a lower-level agent completes its task in the current region and actively reports a termination signal, the higher-level agent is reactivated and selects the next target region. While the lower-level agents are executing, the higher-level agents remain silent.

8. The method for normalized assisted monitoring and optimization of natural resources based on reinforcement learning according to claim 1, characterized in that: In step S6, the results of collaborative decision-making are fused into features and mapped to a continuous state vector. The action space is defined as a continuous parameter increment vector. Simultaneously, an exploration mechanism is introduced, with a pre-defined constrained incremental action generation formula, implemented as follows: ， In the formula, and These represent the system control parameter vector at the current moment and the parameter vector at the next moment after optimization, respectively. The output of the policy network is represented by the current state. Predict a specific direction for basic optimization. This represents random noise that follows a Gaussian distribution. This indicates the intensity of the exploration. This represents the hyperbolic tangent activation function. Basic step size hyperparameter, Indicates the exploration coefficient. This represents a collaborative decision-making mask.

9. The method for normalized assisted monitoring and optimization of natural resources based on reinforcement learning according to claim 1, characterized in that: In step S6, the actual value of incremental actions is accurately evaluated and used to guide the updating of the policy network. This is achieved through value evaluation incorporating the idea of ​​generalized advantage estimation. ， In the formula, This represents the estimated advantage function value for the parameter increment action at the current moment. This represents the instant reward function. and This represents the Critic value network's long-term value assessment of the current state and the next state. Indicates the discount factor. This represents the generalized advantage estimation decay factor, which reduces the advantage value. Parameter convergence status and final optimized parameters Encapsulated as optimization feedback information.

10. The method for normalized assisted monitoring and optimization of natural resources based on reinforcement learning according to claim 1, characterized in that: In step S7, real-time optimization feedback information is collected, and in-depth attribution analysis is performed on optimization results that do not meet expectations. The feedback signal after attribution analysis is used as a reward or penalty signal in reinforcement learning and backpropagated to the Critic value network. The complexity of the current task is evaluated in real time based on the optimization difficulty of the feedback. If the system detects that the change characteristics of the current region are extremely complex, requiring larger-scale exploration or finer boundary fine-tuning, it is determined to be a high-load task. Elastic scaling of computing resources: Based on the evaluation results, the computing resource quotas allocated to high- and low-level agents are dynamically adjusted. The updated strategy and new resource allocation scheme are executed, and their actual performance is closely monitored in the next round. The effectiveness of self-optimization and resource reallocation is verified by comparing key indicators before and after optimization.

Images