A visual language fusion remote sensing change detection method for open pit mine

Abstract

The application provides an open pit mine visual language fusion remote sensing change detection method, relates to the technical field of remote sensing image intelligent interpretation and computer vision, and comprises the following steps: taking double-phase remote sensing images as input, modeling the iteration refinement of change description and ground frame into a Markov decision process through a reinforcement learning agent, and iteratively generating a change semantic description text sequence and a ground frame; visual features are extracted and difference feature maps are calculated through a visual encoder; description texts are encoded into a text feature sequence through a text encoder; the difference feature maps and the text feature sequence are fused through a change perception cross-attention module to obtain fusion features; a pixel-level change mask is generated through a decoder, and finally a triple of a change mask, a change description text and a ground frame is output. Through the iteration self-refining mechanism and change perception fusion, the application realizes interpretable and interactive multi-modal change detection, and improves the small target change detection precision.

Description

Technical Field

[0001] This application relates to the fields of intelligent interpretation of remote sensing images and computer vision technology, and in particular to a method for detecting remote sensing changes in open-pit mines using visual language fusion. Background Technology

[0002] Change detection (CD) is a key technology in the field of Earth observation, widely used in environmental monitoring, urban planning, disaster assessment, and other scenarios. As large-scale resource extraction sites, open-pit mines require high-frequency, large-scale dynamic change monitoring for environmental compliance, safety management, and production optimization. With the integration of high-resolution satellite platforms such as WorldView and Planet with industrial IoT monitoring infrastructure, acquiring large-scale, multi-temporal remote sensing images of mining areas has become feasible, bringing unprecedented opportunities for intelligent, data-driven mine operations. However, existing remote sensing change detection technologies still have significant limitations when used for monitoring open-pit mines.

[0003] Existing single-modal pixel-level change detection methods use dual-temporal remote sensing images as input, relying on visual feature extraction networks based on Siamese convolutional neural networks or Transformer structures (such as representative models like BIT, ChangeFormer, and Changer) to extract visual features from the preceding and following phases. After feature difference or stitching, the decoder outputs a pixel-level change mask. While these methods have achieved certain detection accuracy in general change detection tasks, they rely solely on visual features and do not incorporate textual semantic information. Under complex noise interference such as dust in mining areas, cloud shadows, seasonal vegetation changes, and imaging geometric differences, visual features are easily contaminated, making it difficult to accurately distinguish between genuine mining changes and irrelevant pseudo-changes, resulting in a large number of false positives. Furthermore, their output is only a pixel-level binary mask, unable to answer the semantic question of "what type of change has occurred," and lacks natural language explanations of the causes of the changes. This makes it difficult to support interactive queries and analysis, and fails to meet the actual needs of mining area monitoring for interpretable and interactive results.

[0004] To compensate for the semantic deficiencies of unimodal methods, research has emerged in recent years that introduces visual-language models into change detection tasks. Representative methods include ChangeCLIP, which aligns CLIP text priors with visual features; TextSCD, which injects change descriptions generated by an external model as semantic priors into the decoder; and DeltaVLM, an instruction-guided interactive difference-aware method. However, this type of text-guided multimodal method still has obvious drawbacks: its description generation is one-time and cannot self-correct, it is not conditional on the description and localization results of the previous round, and it lacks a mechanism for iterative refinement in a closed loop. The model cannot observe and correct previously generated illusory descriptions or loose localizations, and its description and localization accuracy for small-scale and small-area changes in mining areas such as equipment movement and step advancement is insufficient; its reinforcement learning reward design is simplistic, only optimizing a single semantic scalar reward, and does not include the intersection-union ratio of the predicted bounding box and the real changed area in the reward function, resulting in the generated semantic text and the real changed area not being effectively aligned in space; its multimodal fusion method is simple, mostly using common methods such as feature splicing, global modulation or standard cross attention, while standard cross attention will also pay attention to the unchanged background area, but the text description only corresponds to the real changed area, causing attention mismatch, which is particularly unfavorable for the detection of small targets and small-area changes. Furthermore, these methods are mostly geared towards general remote sensing scenarios and have not been specifically optimized for the unique characteristics of open-pit mines, such as complex noise, long-tailed variation types, and multi-target coupling. Therefore, there is still considerable room for improvement in detection accuracy and interpretability in actual mine monitoring tasks.

[0005] Existing change detection technologies cannot simultaneously achieve the three-dimensional collaborative output of "which pixels have changed, what has changed, and where has changed" within the same framework. They also lack a self-correcting, spatially grounded descriptive refinement mechanism and a multimodal fusion mechanism sensitive to the changed area. There is an urgent need to propose a new multimodal remote sensing change detection scheme for open-pit mining scenarios to overcome the above-mentioned technical deficiencies. Summary of the Invention

[0006] The purpose of this invention is to address the problems of existing remote sensing change detection methods, which struggle to simultaneously output pixel-level change masks, natural language change descriptions, and change object location boxes within the same framework, and lack self-correcting semantic refinement mechanisms and multimodal fusion mechanisms sensitive to change regions. This invention provides a visual-language fusion remote sensing change detection method for open-pit mines.

[0007] The above-mentioned objective of this application is achieved through the following technical solution: Step S1: Using dual-temporal remote sensing images as input, the iterative refinement of change semantic description and grounding box is modeled as a Markov decision process through the reinforcement learning agent description refinement module, and the change semantic description text sequence and grounding box are iteratively generated and refined. Step S2: Extract visual features from the dual-temporal remote sensing images using a visual encoder and calculate the difference feature map; Step S3: Encode the semantic description text sequence of changes into a text feature sequence using a text encoder; Step S4: Through the change-aware fusion module, the difference feature map and the text feature sequence are fused by change-aware cross-attention to obtain the fused features; Step S5: Input the fused features into the decoder to generate a pixel-level change mask M, and obtain the output triple (M, T, B), where M is the change mask, T is the change semantic description text sequence, and B is the set of change object ground boxes.

[0008] Optionally, step S1 includes: The state space S of the Markov decision process is:

[0009] in To remind us of history, For the first Step description, For the corresponding grounding frame; Generated by initialization of the base VLM using coarse-grained instructions; , This represents the maximum number of refining steps. This refers to earlier time-phase remote sensing images, i.e., remote sensing images acquired at earlier points in time during a change detection task; The latter-temporal remote sensing image refers to the remote sensing image acquired at a later time point in the change detection task, which corresponds pixel-by-pixel after being registered with the former-temporal image; dual-temporal remote sensing images include: former-temporal remote sensing images and latter-temporal remote sensing images. Action space A and state transition T: Action A refined description sequence and coordinates are generated accordingly, by the strategy. The state transition is obtained through VLM parameterized sampling and is a deterministic update:

[0010]

[0011] in, The policy function represents the strategy given the current state. Under certain conditions, the probability distribution of the agent's possible actions; where For the parameters of the policy network, Placeholder for action variable; Represents the state transition function; In the reinforcement learning agent description refinement module, a multi-dimensional composite reward function R is used to evaluate each refinement step. for:

[0012] in: For semantic completeness rewards, BERTScore is used to measure the semantic similarity between generated descriptions and ground truth descriptions, and the improvement of the current step relative to the previous step is used as the reward value. To improve grounding accuracy, the average value is calculated by comparing the intersection and union of predicted grounding frames with the true grounding frames. To provide a fluency reward, the perplexity calculated using the frozen language model is used as a constraint. , , This represents the weighting coefficient for the corresponding reward item; Since the sampling of discrete words and coordinates is not differentiable, a near-end strategy is adopted to optimize the update strategy. Maximize its alternative objectives:

[0013] in Let θ represent the objective function for optimizing the proximal policy, which is used to update the policy parameters θ during reinforcement learning training so that the new policy maximizes the cumulative reward without deviating excessively from the old policy. Represents the mathematical expectation operator, subscript Indicates time step Calculate the expected value of the collected samples; This represents the probability ratio between the old and new strategies. The advantage function is estimated by the generalized advantage estimation. Represents the clipping function; This represents the clipping hyperparameter; The training process is as follows: Initialize the strategy with pre-trained weights ; For each round, by right( ) Perform initial reasoning to obtain and ; right to Loop: Construct state space S; Sample and refine actions get ; calculate , , ; will transfer Store in the buffer; Leveraging the computational advantages of near-end strategies By maximizing Update parameters Continue until convergence, obtaining the optimal strategy. Output a refined sequence of semantic descriptions of changes and a grounding box.

[0014] Optionally, step S2 includes: Visual encoder Regardless of the specific backbone network, the backbone is detected using arbitrary single-modal variation based on CNN or Transformer, and hierarchical features of dual-temporal remote sensing images are extracted. and Then calculate the difference feature map. ;

[0015]

[0016]

[0017] in It can be the difference between absolute values; and These represent the visual features of the preceding and following time phase images, respectively.

[0018] Optionally, step S3 includes: The text encoder uses CLIP Text Tower or BERT; The text encoder encodes the refined semantic description text sequence of changes into a text feature sequence.

[0019] Optionally, step S4 includes: Learnable gated projection Gating is applied to the concatenation result of the difference feature map and the text feature sequence, and the spatial locations of the preceding / following feature pairs that lack information are deweighted to obtain the gated difference query. :

[0020] in For learnable gated projection, σ(·) is the Sigmoid function, ⊙ is element-wise multiplication, and LN is layer normalization; the spatial locations of uninformative features before / after include: unchanged ground and seasonal disturbances unrelated to mining; by For querying, using text feature sequences Given key K and value V, after multi-head cross-attention (MHCA) and residual addition, the fused features are obtained:

[0021] in, The feature map output after fusion by the change-aware cross-attention module integrates the difference information of the two temporal images and the semantic information of the text description, and is used for subsequent decoding to generate a change mask; The learnable projection matrix represents the query. The learnable projection matrix representing the key; The learnable projection matrix representing the value. Optionally, the method integrates any one or more of the following five pluggable multimodal fusion strategies within the same vision-language framework: post-fusion splicing, post-fusion multilayer perceptron, feature-level linear modulation, standard cross attention, and change-aware cross attention.

[0022] An electronic device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to enable the electronic device to perform a visual-language fusion remote sensing change detection method for open-pit mines.

[0023] A computer-readable storage medium storing instructions that, when executed, perform a method for detecting changes in an open-pit mine using visual-language fusion remote sensing.

[0024] The beneficial effects of the technical solution provided in this application are: Iterative self-refinement mechanism: The change description and grounding box are refined and modeled as a Markov decision process. The state explicitly carries the description and grounding box of the previous step, enabling the agent to self-correct in multiple steps in the closed loop and overcome the illusion problem of one-time generation.

[0025] A multidimensional composite reward function couples semantic completeness (BERTScore, relative improvement), grounding precision (IoU), and linguistic fluency (confusion constraint), forcing text descriptions to align with the actual change region space.

[0026] Change-aware cross-attention: By generating difference-guided queries through learnable gated projections, attention is focused on areas of real change, solving the problem that standard cross-attention is sensitive to unchanged backgrounds.

[0027] Unified multimodal fusion framework: Provides five pluggable fusion strategies independent of the visual encoder, enabling collaborative output of mask, description, and bounding box triples. Attached Figure Description

[0028] The present application will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1This is a flowchart illustrating the overall process of the multimodal remote sensing change detection method in this application. Figure 2 This is a refined Markov decision process diagram describing the RL-ACR agent in the embodiments of this application; Figure 3 This is a structural diagram of the change-aware cross-attention (CA-Diff) module in an embodiment of this application; Figure 4 This is a multimodal fusion unified framework diagram of the visual encoder in the embodiments of this application; Figure 5 This is a schematic diagram of the electronic device structure in the embodiments of this application; Figure 6 This is a flowchart of an embodiment of this application; Figure 7 This is a flowchart of the RL-ACR training algorithm in the embodiments of this application; Figure 8 This is a hyperparameter sensitivity analysis diagram from an embodiment of this application. Detailed Implementation

[0029] To provide a clearer understanding of the technical features, objectives, and effects of this application, the specific embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0030] The embodiments of this application provide a method for detecting visual-language fusion remote sensing changes in open-pit mines.

[0031] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the overall process of a multimodal remote sensing change detection method for an open-pit mine area using visual-language fusion remote sensing change detection, as described in this application. The method includes: Step S1: Using dual-temporal remote sensing images as input, the iterative refinement of change semantic description and grounding box is modeled as a Markov decision process through the reinforcement learning agent description refinement module, and the change semantic description text sequence and grounding box are iteratively generated and refined. Step S2: Extract visual features from the dual-temporal remote sensing images using a visual encoder and calculate the difference feature map; Step S3: Encode the semantic description text sequence of changes into a text feature sequence using a text encoder; Step S4: Through the change-aware fusion module, the difference feature map and the text feature sequence are fused by change-aware cross-attention to obtain the fused features; Step S5: Input the fused features into the decoder to generate a pixel-level change mask M, and obtain the output triple (M, T, B), where M is the change mask, T is the change semantic description text sequence, and B is the set of change object ground boxes.

[0032] This application employs the aforementioned technical solution, using a reinforcement learning-driven agent to iteratively refine the semantic description of changes and the grounding box (i.e., the coordinates of the bounding box used to locate the changing object in an image), and then fuses the refined text semantics with visual difference features for change perception, thereby simultaneously outputting a pixel-level change mask, a natural language change description, and a changing object location box in a computer implementation method and system.

[0033] Step S1 includes: The state space S of the Markov decision process is:

[0034] in To remind us of history, For the first Step description, For the corresponding grounding frame; Generated by initialization of the base VLM using coarse-grained instructions; , This is the maximum number of refining steps; This refers to earlier time-phase remote sensing images, i.e., remote sensing images acquired at earlier points in time during a change detection task; The latter-temporal remote sensing image refers to the remote sensing image acquired at a later time point in the change detection task, which corresponds pixel-by-pixel after being registered with the former-temporal image; dual-temporal remote sensing images include: former-temporal remote sensing images and latter-temporal remote sensing images. As one embodiment, the state explicitly carries the "previous step description and grounding frame". This enables intelligent agents to see and correct their previous mistakes.

[0035] Action space A and state transition T: Action A refined description sequence and coordinates are generated accordingly, by the strategy. The state transition is obtained through VLM parameterized sampling and is a deterministic update:

[0036]

[0037] in, The policy function represents the strategy given the current state. Under certain conditions, the probability distribution of the agent's possible actions; where For the parameters of the policy network, Placeholder for action variable; Represents the state transition function; As one example, this mechanism allows the model to self-correct errors in preceding steps, such as refining "vehicle movement" to "excavator movement to the northern area".

[0038] In the reinforcement learning agent description refinement module, a multi-dimensional composite reward function R is used to evaluate each refinement step. for:

[0039] in: For semantic completeness rewards, BERTScore is used to measure the semantic similarity between generated descriptions and ground truth descriptions, and the improvement of the current step relative to the previous step is used as the reward value. To improve grounding accuracy, the average value is calculated by comparing the intersection and union of predicted grounding frames with the true grounding frames. To provide a fluency reward, the perplexity calculated using the frozen language model is used as a constraint. , , This represents the weighting coefficient for the corresponding reward item; As an example, the present invention differs from existing RL-VLM methods in three key ways: First, the refined state carries the description and grounding frame of the previous step, allowing the strategy to be implemented in at most... First, self-correction within the step; second, composite rewards will describe quality. Inter-target grounding IoU ( ( ) Coupling forces the language side to align with the real two-phase changes; third, semantic rewards are calculated relative to the improvement of the previous step rather than absolute scores, penalizing descriptions that fail to improve.

[0040] Since the sampling of discrete words and coordinates is not differentiable, a near-end strategy is adopted to optimize the update strategy. Maximize its alternative objectives:

[0041] in Let θ represent the objective function for optimizing the proximal policy, which is used to update the policy parameters θ during reinforcement learning training so that the new policy maximizes the cumulative reward without deviating excessively from the old policy. Represents the mathematical expectation operator, subscript Indicates time step Calculate the expected value of the collected samples; This represents the probability ratio between the old and new strategies. The advantage function is estimated by the generalized advantage estimation. Represents the clipping function; This represents the clipping hyperparameter; The training process is as follows: Initialize the strategy with pre-trained weights ; For each round, by right( ) Perform initial reasoning to obtain and ; right to Loop: Construct state space S; Sample and refine actions get ; calculate , , ; will transfer Store in the buffer; Leveraging the computational advantages of near-end strategies By maximizing Update parameters ; Repeat until convergence to obtain the optimal strategy. Output a refined sequence of semantic descriptions of changes and a grounding box.

[0042] Figure 6 This is a flowchart of this application; Figure 7 This is the flowchart of the RL-ACR training algorithm.

[0043] Step S2 includes: Visual encoder Regardless of the specific backbone network, the backbone is detected using arbitrary single-modal variation based on CNN or Transformer, and hierarchical features of dual-temporal remote sensing images are extracted. and Then calculate the difference feature map. ;

[0044]

[0045]

[0046] in It can be the difference between absolute values; and These represent the visual features of the preceding and following time phase images, respectively.

[0047] Step S3 includes: The text encoder uses CLIP Text Tower or BERT; The text encoder encodes the refined semantic description text sequence of changes into a text feature sequence.

[0048] Step S4 includes: As one embodiment, Change-Aware Cross-Attention (CA-Diff) addresses the shortcoming of standard cross-attention, which simultaneously focuses on the unchanged background while the text only describes the changed areas. This invention proposes a change-aware fusion mechanism that guides queries using dual-temporal differences. Its structure is shown below. Figure 3 .

[0049] Learnable gated projection Gating is applied to the concatenation result of the difference feature map and the text feature sequence, and the spatial locations of the preceding / following feature pairs that lack information are deweighted to obtain the gated difference query. :

[0050] in For learnable gated projection, σ(·) is the Sigmoid function, ⊙ is element-wise multiplication, and LN is layer normalization; the spatial locations of uninformative features before / after include: unchanged ground and seasonal disturbances unrelated to mining; by For querying, using text feature sequences Given key K and value V, after multi-head cross-attention (MHCA) and residual addition, the fused features are obtained:

[0051] in, The feature map output after fusion by the change-aware cross-attention module integrates the difference information of the two temporal images and the semantic information of the text description, and is used for subsequent decoding to generate a change mask; The learnable projection matrix represents the query. The learnable projection matrix representing the key; The learnable projection matrix representing the value. As one example, change-aware cross-attention (CA-Diff) adds only a small projection compared to standard cross-attention. Despite having a limited number of parameters, it can focus attention on the actual areas of change, thus providing the most significant improvement for small targets and small-area changes. The fused features are decoded to obtain the change mask M.

[0052] The method integrates any one or more of the following five pluggable multimodal fusion strategies within the same vision-language framework: post-fusion splicing, post-fusion multilayer perceptron, feature-level linear modulation, standard cross attention, and change-aware cross attention.

[0053] As one embodiment, to provide a systematically comparable unified framework for multimodal fusion, this invention integrates five pluggable fusion strategies under the same dual-branch structure, categorized by interaction granularity, so that any visual change detection backbone can be included in the same evaluation protocol (see...). Figure 4 ): (1) Dense Interaction: Through Multi-head Cross Attention (MHCA), fine-grained associations between each visual pixel and each text lexical are modeled.

[0054] (2) Global Modulation: FiLM (Feature-wise Linear Modulation): After pooling the text features into a global vector, it is projected to generate scaling γ and translation β. Then perform a channel-by-channel affine transformation. .

[0055] (3) Post-fusion and splicing + MLP: The text is embedded into the global pooling space and copied to H′×W′, along the channel dimension and The data is then stitched together and fused using a multilayer perceptron (MLP) to obtain the fused features. .

[0056] (4) Post-fusion original splicing Concat: directly spliced ​​without MLP, and fused by the decoder implicitly as a baseline.

[0057] (5) Change-aware cross-attention CA-Diff: Based on CA, gated difference query is introduced, which is the best-performing fusion strategy in this framework.

[0058] The present invention also provides a system for implementing the above method, comprising interconnected: a description refining unit (performing RL-ACR), a visual encoding unit, a text encoding unit, a change-aware fusion unit (performing CA-Diff or any of the above fusion strategies), and a decoding output unit; the system is deployed on a computing device with a graphics processing unit (GPU) and can also operate as a service for monitoring mining areas for the Industrial Internet of Things (IoT).

[0059] In one embodiment, the implementation details are as follows: (1) Data and preprocessing: Input dual-phase images were uniformly scaled to 256×256 pixels and normalized according to ImageNet mean and standard deviation; training employed data augmentation methods such as random horizontal flip (probability 0.5), random vertical flip (probability 0.5), and ±15° random rotation.

[0060] (2) RL-ACR: Policy Network Initialize with InstructBLIP (using Instructblip-Vicuna-7B checkpoints); use Vicuna-7B text encoder; PPO hyperparameters are clipping ratio ε=0.2, discount factor γ=0.99, and GAE parameter λ=0.95; maximum refining steps. =5; Reward Weight =0.4、 =0.4、 =0.2; Smoothness threshold τ=20.0, see Figure 8 .

[0061] (3) Refinement Hints (Thought Chain): In each refinement step, guide the agent according to the following thought chain strategy: ① Verify Existence (check whether the objects in the description actually exist and have changed, and remove illusions); ② Check Completeness (supplement visible but omitted changes, such as new roads or the movement of specific machinery); ③ Refine Terminology (upgrade general terms to specialized mining terms, such as "pit" → "mining bench"); ④ Adjust Coordinates (tighten the bounding box to accurately enclose the changing pixels). The output is in JSON format, containing a list of reasoning, refined_caption, and grounding, where each grounding contains the object and box coordinates.

[0062] (4) Coordinate normalization: Normalize all coordinates to the interval [0,1000], with the format [ymin,xmin,ymax,xmax], the upper left corner is [0,0] and the lower right corner is [1000,1000]; each change mentioned in the description must have a corresponding bounding box.

[0063] (5) Change detection branch training: The text embedding uses pre-trained BERT; the optimizer is AdamW, and the learning rate is 1×10. -4 Weight decay 1×10 -4 It employs a cosine annealing learning rate with 5 rounds of linear warm-up; batch size 16; training 100 rounds; and can be trained on multi-GPU systems with mixed precision (FP16).

[0064] (6) Fusion strategy selection: CA-Diff or CA is preferred in mining area monitoring where high accuracy is required; FiLM or Concat can be selected in scenarios where real-time requirements are high to reduce latency.

[0065] The workflow of this embodiment is as follows: inputting dual-temporal images into RL-ACR to obtain refined description C and ground frame B ( Figure 2 ); obtained through a visual encoder and And calculate The result obtained by the text encoder ; via CA-Diff ( Figure 3 After fusion, the decoder outputs a transformed mask M; the final output is a triplet (M, T, B). Figure 1 ) In one embodiment, the single-modal technique has the following technical disadvantages: Disadvantage 1: Single-modal constraint: It only utilizes the visual features of dual-temporal images and does not introduce the enhancement effect of textual semantics on the description of changes; under the complex noise interference such as dust, shadows, and seasonal vegetation changes in mining areas, visual features are easily contaminated, leading to a large number of false detections (false positives).

[0066] Disadvantage 2: Semantic deficiency: The output is only a pixel-level change mask, which cannot directly answer the semantic question of "what type of change has occurred (such as pit expansion, spoil heap growth, equipment movement, vegetation restoration, etc.)".

[0067] Disadvantage 3: Insufficient interpretability: It lacks natural language explanations for the causes of change, cannot support human-computer interactive decision-making, and is difficult to meet the needs of actual business operations such as environmental compliance, safety governance, and production optimization for interactive and interpretable results.

[0068] In one embodiment, text-guided / multimodal technology still suffers from the following technical drawbacks: Disadvantage 4: Description generation is one-time and cannot self-correct: Existing methods either rely on manual templates (such as the prompt templates of ChangeCLIP) or rely on fixed pre-trained descriptors to generate text once (such as TextSCD). The generation process is not based on the "previous round of description and localization box", so the model cannot see its previous errors and cannot iteratively correct hallucinations (i.e., describing objects that do not exist in the image) and loose localizations in the closed loop.

[0069] Disadvantage 5: Single reward and lack of coupling with spatial grounding: Existing methods that use reinforcement learning to describe generation (such as VLRM with VLM as the reward model, SC-Captioner with single-step self-correction, and general RLHF-CIDEr, etc.) only optimize a single semantic scalar reward and do not directly incorporate the spatial grounding accuracy of the bounding box (such as the intersection-union ratio IoU) into the reward, resulting in spatial misalignment between the text and the real change area.

[0070] Disadvantage 6: Single-pass reasoning without multi-step refinement: The above methods are all single-pass reasoning and lack a multi-step iterative refinement mechanism, which is insufficient for describing and locating small-scale and small-area changes such as "equipment movement" in open-pit mines.

[0071] Disadvantage 7: The fusion method is not systematic and is not sensitive to changes: Existing multimodal fusion mostly adopts simple splicing or global modulation. Standard cross attention treats the visual side as a single feature stream and will pay attention to the background area that has not changed at the same time, while the text only describes the changed area, resulting in attention mismatch, which is particularly unfavorable for the detection of small targets and small area changes.

[0072] Existing technologies cannot simultaneously achieve the coordinated output of "variable mask, natural language description, and target grounding box" within the same framework, and lack a description refinement mechanism that can self-correct and be coupled with spatial grounding, as well as a multimodal fusion mechanism that is sensitive to changing regions.

[0073] The beneficial effects of the technical solution provided in this application are: 1. Triple-based collaborative output enhances interpretability and interactivity: Simultaneously outputting the change mask M, natural language change description T, and change object grounding box B within the same framework, the monitoring results can answer "which pixels changed, what changed, and where the change occurred", supporting human-computer interactive decision-making and overcoming shortcomings 2 and 3.

[0074] 2. Textual semantics acts as a "semantic filter," significantly reducing false positives: After introducing textual descriptions, detection performance is consistently improved across all backbones and all fusion strategies, and the improvement in precision is significantly greater than that in recall. For example, cross-attention on a certain backbone increases recall by about 1.53% while increasing precision by about 9.74%, indicating that text is mainly used to suppress false positives caused by seasonal vegetation, dust, shadows, etc., overcoming the shortcoming 1.

[0075] 3. The finer the fusion granularity, the greater the gain, and the stronger the compensation for weak backbones: the performance gain shows the hierarchy of CA (Dense Interaction) > FiLM (Global Modulation) > MLP / Concat (Post-fusion); at the same time, the relative gain obtained by weaker backbones is greater (text acts as a "knowledge compensator"), and the absolute improvement of F1 on a certain lightweight backbone can reach about +3.97%.

[0076] 4. CA-Diff further narrows the performance gap, especially beneficial for small targets / small area changes: On the strongest backbone, CA-Diff still brings an F1 improvement of about +1.60 compared to the already strong CA; in small target scenarios, it improves by about +1.75 compared to CA, and the gain is the greatest in seasonal vegetation scenarios, which verifies the role of gating difference query in focusing attention on the real change area and overcoming the shortcomings of 7.

[0077] 5. RL-ACR outperforms large models with small models, and RL refinement brings significant leaps: the approximately 7B parameter domain model refined by RL-ACR surpasses general-purpose large models with more than 30 times the number of parameters in terms of description quality; compared to the same model without reinforcement learning, key description quality metrics (such as METEOR, CIDEr) and grounding accuracy (IoU, mAP@0.5) all monotonically improve and... The peak value was reached, proving that semantic completeness reward and grounding accuracy reward can effectively guide the model to generate descriptions that are fluent, factually accurate and spatially aligned, overcoming shortcomings 4, 5 and 6.

[0078] 6. Robust generalization to geography and climate: The proposed method exhibits moderate performance dispersion under various conditions across continents, climate zones, and terrains. The average performance after weighting is basically consistent with the global performance, indicating that the scheme has good engineering generalization and practicality.

[0079] This application also discloses an electronic device. (See reference...) Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. The electronic device 500 may include: at least one processor 501, at least one network interface 504, a user interface 503, a memory 505, and at least one communication bus 502.

[0080] The communication bus 502 is used to enable communication between these components.

[0081] The user interface 503 may include a display screen, and optionally, the user interface 503 may also include a standard wired interface or a wireless interface.

[0082] The network interface 504 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0083] This application also discloses a computer-readable storage medium storing multiple instructions adapted for loading by a processor to execute the above-described method for detecting visual-language fusion remote sensing changes in open-pit mines.

[0084] The above are merely exemplary embodiments of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure.

[0085] This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

Claims

1. A method for detecting visual language fusion remote sensing changes in open-pit mines, characterized in that, The method includes the following steps: Step S1: Using dual-temporal remote sensing images as input, the iterative refinement of change semantic description and grounding box is modeled as a Markov decision process through the reinforcement learning agent description refinement module, and the change semantic description text sequence and grounding box are iteratively generated and refined. Step S2: Extract visual features from the dual-temporal remote sensing images using a visual encoder and calculate the difference feature map; Step S3: Encode the semantic description text sequence of changes into a text feature sequence using a text encoder; Step S4: Through the change-aware fusion module, the difference feature map and the text feature sequence are fused by change-aware cross-attention to obtain the fused features; Step S5: Input the fused features into the decoder to generate a pixel-level change mask M, and obtain the output triple (M, T, B), where M is the change mask, T is the change semantic description text sequence, and B is the set of change object ground boxes.

2. The method for detecting visual-language fusion remote sensing changes in open-pit mines as described in claim 1, characterized in that, Step S1 includes: The state space S of the Markov decision process is: in To remind us of history, For the first Step description, For the corresponding grounding frame; Generated by initialization of the base VLM using coarse-grained instructions; , This represents the maximum number of refining steps. This refers to earlier time-phase remote sensing images, i.e., remote sensing images acquired at earlier points in time during a change detection task; The latter-temporal remote sensing image refers to the remote sensing image acquired at a later time point in the change detection task, which corresponds pixel-by-pixel after being registered with the former-temporal image; dual-temporal remote sensing images include: former-temporal remote sensing images and latter-temporal remote sensing images. Action space A and state transition T: Action A refined description sequence and coordinates are generated accordingly, by the strategy. The state transition is obtained through VLM parameterized sampling and is a deterministic update: in, The policy function represents the strategy given the current state. Under certain conditions, the probability distribution of the agent's possible actions; where For the parameters of the policy network, Placeholder for action variable; Represents the state transition function; In the reinforcement learning agent description refinement module, a multi-dimensional composite reward function R is used to evaluate each refinement step. for: in: For semantic completeness rewards, BERTScore is used to measure the semantic similarity between generated descriptions and ground truth descriptions, and the improvement of the current step relative to the previous step is used as the reward value. To improve grounding accuracy, the average value is calculated by comparing the intersection and union of predicted grounding frames with the true grounding frames. To provide a fluency reward, the perplexity calculated using the frozen language model is used as a constraint. , , This represents the weighting coefficient for the corresponding reward item; Since the sampling of discrete words and coordinates is not differentiable, a near-end strategy is adopted to optimize the update strategy. Maximize its alternative objectives: in Let θ represent the objective function for optimizing the proximal policy, which is used to update the policy parameters θ during reinforcement learning training so that the new policy maximizes the cumulative reward without deviating excessively from the old policy. Represents the mathematical expectation operator, subscript Indicates time step Calculate the expected value of the collected samples; This represents the probability ratio between the old and new strategies. The advantage function is estimated by the generalized advantage estimation. Represents the clipping function; This represents the clipping hyperparameter; The training process is as follows: Initialize the strategy with pre-trained weights ; For each round, by right( ) Perform initial reasoning to obtain and ; right to Loop: Construct state space S; Sample and refine actions get ; calculate , , ; will transfer Store in the buffer; Leveraging the computational advantages of near-end strategies By maximizing Update parameters Continue until convergence, obtaining the optimal strategy. Output a refined sequence of semantic descriptions of changes and a grounding box.

3. The method for detecting visual-language fusion remote sensing changes in open-pit mines as described in claim 1, characterized in that, Step S2 includes: Visual encoder Regardless of the specific backbone network, the backbone is detected using arbitrary single-modal variation based on CNN or Transformer, and hierarchical features of dual-temporal remote sensing images are extracted. and Then calculate the difference feature map. ; in It can be the difference between absolute values; and These represent the visual features of the preceding and following time phase images, respectively.

4. The method for detecting visual-language fusion remote sensing changes in open-pit mines as described in claim 1, characterized in that, Step S3 includes: The text encoder uses CLIP Text Tower or BERT; The text encoder encodes the refined semantic description text sequence of changes into a text feature sequence.

5. The method for detecting visual-language fusion remote sensing changes in open-pit mines as described in claim 3, characterized in that, Step S4 includes: Learnable gated projection Gating is applied to the concatenation result of the difference feature map and the text feature sequence, and the spatial locations of the preceding / following feature pairs that lack information are deweighted to obtain the gated difference query. : in For learnable gated projection, σ(·) is the Sigmoid function, ⊙ is element-wise multiplication, and LN is layer normalization; the spatial locations of uninformative features before / after include: unchanged ground and seasonal disturbances unrelated to mining; by For querying, using text feature sequences Given key K and value V, after multi-head cross-attention (MHCA) and residual addition, the fused features are obtained: in, The feature map output after fusion by the change-aware cross-attention module integrates the difference information of the two temporal images and the semantic information of the text description, and is used for subsequent decoding to generate a change mask; The learnable projection matrix represents the query. The learnable projection matrix representing the key; The learnable projection matrix representing the value.

6. The method for detecting visual-language fusion remote sensing changes in open-pit mines as described in claim 3, characterized in that, The method integrates any one or more of the following five pluggable multimodal fusion strategies within the same vision-language framework: post-fusion splicing, post-fusion multilayer perceptron, feature-level linear modulation, standard cross attention, and change-aware cross attention.

7. An electronic device, characterized in that, The device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to enable the electronic device to perform the visual-language fusion remote sensing change detection method for open-pit mines as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed by a computer, perform the open-pit mine visual-language fusion remote sensing change detection method as described in any one of claims 1-6.

Images