Remote sensing image semantic segmentation method based on context joint reasoning

Abstract

The invention relates to a remote sensing image semantic segmentation method based on context joint reasoning, and belongs to the field of remote sensing image processing and computer vision. Comprising the steps that semantic prior probability distribution is obtained through a visual language model; obtaining a visual feature set of the whole scene through the visual basic model, fitting GMM on the scene, and obtaining context posterior distribution; introducing KL divergence to construct a dual-target alignment function, and performing joint optimization on semantic prior and context posteriori; by alternately optimizing common convergence of GMM parameters, a context model gradually absorbs semantic alignment information, and semantic prediction is constrained and enhanced by a scene structure; and directly outputting the calibrated patch semantic probability during reasoning, and obtaining a final segmentation mask of the large-scale remote sensing scene through position mapping and up-sampling reconstruction full-resolution dense segmentation. According to the method, the practicability and the robustness of the VLM-based open vocabulary remote sensing segmentation method in a complex and large-scale scene are remarkably enhanced.

Description

Technical Field

[0001] This invention relates to a semantic segmentation method for remote sensing images based on contextual joint reasoning, belonging to the fields of remote sensing image processing and computer vision. Background Technology

[0002] Semantic segmentation is a core task in remote sensing image interpretation, providing crucial technical support for applications such as land cover mapping, urban planning, and environmental monitoring. Traditional remote sensing image segmentation methods are typically developed based on the closed-set assumption, meaning the model can only recognize a fixed set of categories predefined during training. This setting struggles to handle the frequent occurrence of unknown categories in large-scale, heterogeneous real-world environments, thus prompting the exploration of "open-vocabulary remote sensing image segmentation" technology. However, most existing training-free open-vocabulary remote sensing image segmentation methods directly adopt the processing paradigm of these natural images, mainly following two research directions: one is to enhance the visual feature representation of image patches to improve their discriminative ability for dense prediction tasks; the other is to strive to reduce the modal differences between visual features and textual prototypes to improve the consistency of cross-modal semantic alignment. These existing methods largely inherit the "independent prediction" paradigm, treating each image patch as an isolated unit while ignoring the fundamental characteristics inherent in remote sensing data. Unlike natural images, remote sensing images typically cover a vast geographical area, exhibiting complex spatial organization and strong semantic correlations. Furthermore, visually similar regions, such as roads and runways, may carry drastically different semantic information due to their different spatial contexts. Therefore, prediction schemes based purely on isolated image patches are prone to fragmentation and misclassification in segmentation results, making it difficult to guarantee segmentation consistency and global rationality across a wide range of scenes. In summary, this invention proposes a context-aware reasoning framework that effectively incorporates contextual information to capture spatial dependencies, improving segmentation consistency and robustness without requiring fine-tuning of pre-trained models or relying on large amounts of labeled data. Summary of the Invention

[0003] To overcome the limitations of current methods, this invention provides a remote sensing semantic segmentation method based on contextual joint reasoning. This method addresses the fragmentation and category confusion in segmentation results caused by the large-scale spatial organization and strong semantic correlation of remote sensing images. This invention bridges the gap between semantic understanding and spatial reasoning, significantly improving the consistency, robustness, and generalization ability of segmentation in complex remote sensing environments. Furthermore, it consistently outperforms existing untrained and state-of-the-art VLM-based baselines on various remote sensing datasets.

[0004] The technical solution of this invention is: a semantic segmentation method for remote sensing images based on contextual joint reasoning, the method comprising:

[0005] Step 1: Divide the large-scale remote sensing scene into several images of fixed size. Each image patch will be processed independently to extract two types of patch-level features.

[0006] Step 2: Use the image encoder in the visual language model to divide each image block into patches and extract visual features. Use the text encoder to extract semantic information from the text prompts with embedded categories. Calculate the similarity between the visual features and the text features to obtain the prior probability of the category.

[0007] Step 3: Extract contextual information from the large visual model. Input each image patch into the basic visual model to extract visual features with robust spatial structure. By fitting a Gaussian mixture model onto the remote sensing scene, the contextual posterior probability distribution of the image is obtained.

[0008] Step 4: Construct a joint optimization objective function. First, two types of information are introduced as constraints: one is the semantic prior from the visual language model, which provides a category probability distribution based on the text prototype for each image patch; the other is the contextual posterior obtained from the visual base model across the entire scene, which introduces KL divergence to measure the distribution difference and constrains the degree of deviation between the two, thereby organizing semantic information and structural information into a unified optimization objective.

[0009] Step 5: Alternately optimize the prediction distribution and Gaussian Mixture Model (GMM) parameters using the EM algorithm. First, calculate the context posterior for each patch based on the current structural model. Then, under the condition of fixing the posterior, obtain the calibrated patch-level prediction by fusing the semantic prior and the context posterior. Then, use these calibrated predictions as soft responsibility to update the structural model parameters. Iterate until the target converges, so that visual clustering and language categories are better aligned, and semantic prediction is subject to global space constraints, thereby suppressing isolated or fragmented erroneous predictions. After iterative convergence, output the calibrated semantic probability distribution for each patch.

[0010] Step 6, Reasoning and Reconstruction: The calibration patch probabilities obtained after optimization and convergence are mapped back to the full scene according to their original positions, and a large-scale remote sensing scene final segmentation mask with full resolution and spatial consistency is generated through upsampling.

[0011] Further, Step 1 includes:

[0012] The large-scale remote sensing scene is divided into several image patches of fixed pixel size according to the principle of non-overlapping. Each image patch is input into the image encoder in the visual base model DINOv3 and the visual language model respectively, and the patches of each image patch are processed to extract features.

[0013] Furthermore, Step 2 specifically includes the following steps:

[0014] Step 2.1: For the visual language model branch, input the processed image patches into the image encoder to extract patch-level visual features, and input the category cue template into the text encoder to obtain the category prior probability. Specifically, this includes: first, setting the number of segments and patches; the size of each image patch is W×H, and the pixel size of the patch is p×p; then the number of patches per tile is... Visual encoder using VLM Forward reasoning is performed on each patch to obtain a patch-level visual representation:

[0015]

[0016] Where n represents the number of patches in each image patch, and d represents the feature dimension; simultaneously, given a set of candidate categories C is a user-defined set of candidate categories. For each category, a text cue is constructed, such as "a satellite image of a {class}", and a text encoder is used. The encoded text prototype vector set T can be represented as:

[0017] }

[0018] Step 2.2: Using normalized cosine similarity as a measure of semantic relevance between the patch and the original text, the similarity score is calculated, in the form of:

[0019]

[0020] in, Category-specific prompt templates, For visual feature vectors, For text feature vectors, It is a temperature hyperparameter;

[0021] The similarity scores are converted into patch-level class probabilities, scaled using a temperature parameter, and then softmax normalized along the class dimension to obtain the semantic prior distribution, whose components are represented as follows:

[0022]

[0023] The resulting patch-level semantic prior set is used as the semantic constraint input for the subsequent context joint reasoning module. Together with the context posterior obtained by fitting the visual base model across the entire scene, it participates in joint optimization to achieve semantic and spatial consistency calibration and output the final calibration prediction.

[0024] Furthermore, the specific steps for extracting contextual information from the remote sensing image in Step 3 include:

[0025] By using the visual foundation model DINOv3 to extract high-level visual features from all patches of the entire large-scale scene to form a patch set, the contextual dependencies in the remote sensing image are captured, which can be represented as: To characterize the contextual structure and visual clustering relationships within a scene, a Gaussian Mixture Model (GMM) with k mixture components is fitted onto the feature set X, and its density is expressed as:

[0026]

[0027] in, Let be the mixing weights, mean vector, and covariance of the k-th component, respectively. Let the number of mixing components k be the same as the size of the candidate category set in the visual language branch, i.e., k = |C|. Based on the current GMM parameters, calculate the contextual posterior distribution of each patch feature in each mixing component:

[0028]

[0029] Furthermore, Step 4 includes:

[0030] For the i-th patch in the remote sensing scene, a potential consistency probability distribution is introduced. As the latent variable to be determined, a prior probability distribution is defined. posterior probability distribution of visual context First, we construct a patch-by-patch consistency minimization objective based on prior and posterior a priori assumptions, formalized as:

[0031]

[0032] But due to The context model, derived from estimation in the visual feature space, does not naturally have semantic priors. Alignment, therefore optimizing the above objective alone is only seeking... Limited value. To overcome this problem, this invention proposes a method for analyzing the potential consensus distribution { Joint learning with GMM parameters allows for contextual posterior learning. As the optimization process gradually incorporates semantic alignment information, the two reinforce each other, and the joint optimization objective is denoted as:

[0033]

[0034] because Follow Σ changes with updates, when certain patches are released. When weak or noisy, the context It can enhance connectivity; conversely, when visual clustering confuses semantics, semantic priors can guide the Gaussian Mixture Model (GMM) to re-divide, thereby reducing category confusion and fragmentation, and reducing errors caused by semantic-visual mismatch.

[0035] Furthermore, Step 5 employs an alternating optimization strategy based on expectation maximization (EM), alternately updating the GMM parameters and the patch-level consistency distribution to ensure contextual posteriority. With potential consensus distribution The mutual reinforcement and gradual convergence are achieved through the following specific steps:

[0036] Step 5.1: Initialize key parameters, starting with the semantic prior distribution from the visual language branch. Initialize the mean and covariance of the Gaussian mixture model, and simultaneously set the mixture weights. Keep it fixed as a uniform distribution , where k is the number of components in the Gaussian mixture model, which is the same as the number of candidate classes. This can be formalized as:

[0037]

[0038]

[0039]

[0040] in, This represents the probability that the i-th patch belongs to the k-th Gaussian mixture model component;

[0041] Step 5.2: Alternating Iterative Updates. After the above initialization, execute the following three steps alternately until convergence or the preset maximum number of iterations is reached:

[0042] (1) E-step: Update the context posterior q, in the current Gaussian mixture model parameters { The responsibility of each patch on each component is calculated posteriorly according to GMM, in the following form:

[0043]

[0044] (2) Z-update: Update the potential consistency distribution z. Given a fixed semantic prior p and a current context posterior q, obtain the analytical update for each patch according to the KL consistency objective:

[0045] ;

[0046] ;

[0047] Among them, the product operation emphasizes the consistency between 𝑝 and q. 𝑧 is high only when both probabilities are high. If one of them is low, z is suppressed, thus realizing the fusion of prior knowledge and contextual information and enhancing the reliability of understanding.

[0048] (3) M-step: The updated z is regarded as the soft responsibility of this M-step, and the mean and covariance of the GMM are re-estimated accordingly. The specific update formula is as follows:

[0049]

[0050]

[0051] Alternatingly executing the above three steps E, Z, and M decreases the objective function, thereby achieving gradual alignment and mutual reinforcement of z and q, ultimately leading to convergence of { } was used as a calibrated patch-level semantic distribution to generate the final segmentation result.

[0052] Furthermore, the inference and reconstruction package in Step 6 optimizes the converged calibration patch probabilities, maps them back to the full scene based on their original positions, and generates a full-resolution, spatially consistent large-scale remote sensing scene final segmentation mask through upsampling. The specific steps include:

[0053] Step 6.1, Location Mapping: Based on the patch spatial location recorded during image segmentation, the patch-level probability vector is precisely placed in the corresponding region of the global probability map to form a seamless but low-resolution (patch-level) initial segmentation probability map.

[0054] Step 6.2, Probabilistic Upsampling: Using spatial consistency preservation methods such as bilinear interpolation, the initial probability map is upsampled to restore the full resolution of the original input image. This process fully respects and preserves the scene structural context enhanced by GMM modeling, allowing semantic boundaries to naturally conform to the overall trend of ground feature distribution during the refinement process;

[0055] Step 6.3: Generate the final mask: Perform an argmax operation on the full-resolution probability map for each pixel to determine its final semantic category, thereby generating a spatially coherent and detailed large-scale remote sensing scene semantic segmentation mask. Through this front-end optimization, the final output effectively suppresses fragmentation noise caused by misjudgments of isolated patches and significantly improves the boundary consistency and regional integrity of various land features in complex scenes.

[0056] The present invention also provides a remote sensing semantic segmentation system based on context joint reasoning, the system comprising: a module for executing the remote sensing semantic segmentation method based on context joint reasoning.

[0057] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the remote sensing image semantic segmentation method based on context joint reasoning.

[0058] The beneficial effects of this invention are:

[0059] 1. Improve segmentation consistency and accuracy: By jointly calibrating the semantic prior of VLM with the scene context of VFM at the inference stage, a more robust patch-level consistency distribution is obtained, thereby alleviating the common fragmentation and category confusion problems of remote sensing images. The joint optimization objective is formalized to ensure that the final prediction simultaneously considers semantic alignment and spatial structure.

[0060] 2. Free and easy to integrate: The method runs without modifying or fine-tuning any pre-trained model parameters. As an inference-level plugin, it can be seamlessly integrated with existing VLMs (such as CLIP) and VFMs (such as DINOv3), demonstrating good versatility and engineering usability. Large-scale experiments show an average improvement of approximately 2.80% (mIoU) and 6.13% (Foreground IoU) on multiple remote sensing datasets, proving the practical utility of the method.

[0061] 3. Structured Target Extraction and Robustness: By fitting a GMM to the full-scene VFM features and using alternating EM iterations and consistent updates to mutually enhance semantic and contextual structures, the product-type fusion update z in each update effectively recovers targets with structural continuity, such as roads and buildings, and reduces breaks. Fast Convergence, Computational Controllability, and Easy Parallelization: Alternating iterations typically converge within fewer than 10 iterations, and each patch update can be matrix-based and batch-processed, thus maintaining moderate computational overhead while adapting to practical deployments in large-scale remote sensing scenes. These technical features collectively improve the stability, robustness, and generalization ability of the method in real-world remote sensing applications. Attached Figure Description

[0062] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0063] Example 1: As Figure 1 As shown, this invention provides a remote sensing semantic segmentation method based on contextual joint reasoning to address the fragmentation and category confusion in segmentation results caused by the large-scale spatial organization and strong semantic correlation of remote sensing images. This invention bridges the gap between semantic understanding and spatial reasoning, significantly improving the consistency, robustness, and generalization ability of segmentation in complex remote sensing environments. Furthermore, it consistently outperforms existing no-training and state-of-the-art VLM-based baselines on various remote sensing datasets. The method includes:

[0064] Step 1: Divide the large-scale remote sensing scene into several images of fixed size. Each image patch will be processed independently to extract two types of patch-level features.

[0065] Further, Step 1 includes:

[0066] The large-scale remote sensing scene is divided into several image patches of fixed pixel size according to the principle of non-overlapping. Each image patch is input into the image encoder in the visual base model DINOv3 and the visual language model respectively, and the patches of each image patch are processed to extract features.

[0067] Step 2: Use the image encoder in the visual language model to divide each image block into patches and extract visual features. Use the text encoder to extract semantic information from the text prompts with embedded categories. Calculate the similarity between the visual features and the text features to obtain the prior probability of the category.

[0068] Furthermore, Step 2 specifically includes the following steps:

[0069] Step 2.1: For the visual language model branch, input the processed image patches into the image encoder to extract patch-level visual features, and input the category cue template into the text encoder to obtain the category prior probability. Specifically, this includes: first, setting the number of segments and patches; the size of each image patch is W×H, and the pixel size of the patch is p×p; then the number of patches per tile is... Visual encoder using VLM Forward reasoning is performed on each patch to obtain a patch-level visual representation:

[0070]

[0071] Where n represents the number of patches in each image patch, and d represents the feature dimension; simultaneously, given a set of candidate categories C is a user-defined set of candidate categories. For each category, a text cue is constructed, such as "a satellite image of a {class}", and a text encoder is used. The encoded text prototype vector set T can be represented as:

[0072] }

[0073] Step 2.2: Using normalized cosine similarity as a measure of semantic relevance between the patch and the original text, the similarity score is calculated, in the form of:

[0074]

[0075] in, For example, if category c is a tree, then... It will be converted into a prompt template like "a photo of a tree". The visual feature vector is the feature of the i-th patch extracted from the input image by a visual encoder (such as ViT in CLIP). It represents the visual content (such as color, shape, texture) at that location in the image. The text feature vector is generated by placing the category name (such as "tree") into a prompt word template (such as "a photo of a [class]"). It represents the abstract semantic definition of the category. It is a temperature hyperparameter used to adjust the "concentration" of the probability distribution: when When the distribution is smaller, it becomes more "sharp," making the model more confident in the class with the highest similarity. When the value is large, the distribution becomes smoother, and the probability differences between categories decrease;

[0076] The similarity scores are converted into patch-level class probabilities, scaled using a temperature parameter, and then softmax normalized along the class dimension to obtain the semantic prior distribution, whose components are represented as follows:

[0077]

[0078] The resulting patch-level semantic prior set is used as the semantic constraint input for the subsequent context joint reasoning module. Together with the context posterior obtained by fitting the visual base model across the entire scene, it participates in joint optimization to achieve semantic and spatial consistency calibration and output the final calibration prediction.

[0079] Step 3: Extract contextual information from the large visual model. Input each image patch into the basic visual model to extract visual features with robust spatial structure. By fitting a Gaussian mixture model onto the remote sensing scene, the contextual posterior probability distribution of the image is obtained.

[0080] Furthermore, the specific steps for extracting contextual information from remote sensing images in Step 3 include: using the visual foundation model DINOv3 to extract high-level visual features from all patches of the entire large-scale scene to form a patch set, capturing the contextual dependencies in the remote sensing image, which can be represented as: To characterize the contextual structure and visual clustering relationships within a scene, a Gaussian Mixture Model (GMM) with k mixture components is fitted onto the feature set X, and its density is expressed as:

[0081]

[0082] in, Let be the mixing weights, mean vector, and covariance of the k-th component, respectively. Let the number of mixing components k be the same as the size of the candidate category set in the visual language branch, i.e., k = |C|. Based on the current GMM parameters, calculate the contextual posterior distribution of each patch feature in each mixing component:

[0083]

[0084] Step 4: Construct a joint optimization objective function. First, two types of information are introduced as constraints: one is the semantic prior from the visual language model, which provides a category probability distribution based on the text prototype for each image patch; the other is the contextual posterior obtained from the visual base model across the entire scene, which introduces KL divergence to measure the distribution difference and constrains the degree of deviation between the two, thereby organizing semantic information and structural information into a unified optimization objective.

[0085] Furthermore, Step 4 includes: for the i-th patch in the remote sensing scene, introducing a potential consistency probability distribution. As the latent variable to be determined, a prior probability distribution is defined. posterior probability distribution of visual context First, we construct a patch-by-patch consistency minimization objective based on prior and posterior a priori assumptions, formalized as:

[0086]

[0087] But due to The context model, derived from estimation in the visual feature space, does not naturally have semantic priors. Alignment, therefore optimizing the above objective alone is only seeking... Limited value. To overcome this problem, this invention proposes a method for analyzing the potential consensus distribution { Joint learning with GMM parameters allows for contextual posterior learning. As the optimization process gradually incorporates semantic alignment information, the two reinforce each other, and the joint optimization objective is denoted as:

[0088]

[0089] because Follow Σ changes with updates, when certain patches are released. If there is a lot of noise, it comes from the context. It can enhance connectivity; conversely, when visual clustering confuses semantics, semantic priors can guide GMM to re-divide, thereby reducing category confusion and fragmentation, and reducing errors caused by semantic-visual mismatch.

[0090] Step 5: Alternately optimize the prediction distribution and Gaussian Mixture Model (GMM) parameters using the EM algorithm. First, calculate the context posterior for each patch based on the current structural model. Then, under the condition of fixing the posterior, obtain the calibrated patch-level predictions by fusing the semantic prior and context posterior. Next, use these calibrated predictions as soft responsibility to update the structural model parameters. Iterate until the target converges, so that visual clustering and language categories are better aligned, and semantic prediction is subject to global space constraints, thereby suppressing isolated or fragmented erroneous predictions. After iterative convergence, output the calibrated semantic probability distribution for each patch.

[0091] Furthermore, Step 5 employs an alternating optimization strategy based on expectation maximization (EM), alternately updating the GMM parameters and the patch-level consistency distribution to ensure contextual posteriority. With potential consensus distribution The mutual reinforcement and gradual convergence are achieved through the following specific steps:

[0092] Step 5.1: Initialize key parameters, starting with the semantic prior distribution from the visual language branch. Initialize the mean and covariance of the Gaussian mixture model, and simultaneously set the mixture weights. Keep it fixed as a uniform distribution , where k is the number of components in the Gaussian mixture model, which is the same as the number of candidate classes. This can be formalized as:

[0093]

[0094]

[0095]

[0096] in, This represents the probability that the i-th patch belongs to the k-th Gaussian mixture model component;

[0097] Step 5.2: Alternating Iterative Updates. After the above initialization, execute the following three steps alternately until convergence or the preset maximum number of iterations is reached:

[0098] (1) E-step: Update the context posterior q, in the current Gaussian mixture model parameters { The responsibility of each patch on each component is calculated posteriorly according to GMM, in the following form:

[0099]

[0100] (2) Z-update: Update the potential consistency distribution z. Given a fixed semantic prior p and a current context posterior q, obtain the analytical update for each patch according to the KL consistency objective:

[0101] ;

[0102] ;

[0103] Among them, the product operation emphasizes the consistency between 𝑝 and q. 𝑧 is high only when both probabilities are high. If one of them is low, z is suppressed, thus realizing the fusion of prior knowledge and contextual information and enhancing the reliability of understanding.

[0104] (3) M-step: The updated z is regarded as the soft responsibility of this M-step, and the mean and covariance of the GMM are re-estimated accordingly. The specific update formula is as follows:

[0105]

[0106]

[0107] Alternatingly executing the above three steps E, Z, and M decreases the objective function, thereby achieving gradual alignment and mutual reinforcement of z and q, ultimately leading to convergence of { } was used as a calibrated patch-level semantic distribution to generate the final segmentation result.

[0108] Step 6, Reasoning and Reconstruction: The calibration patch probabilities obtained after optimization and convergence are mapped back to the full scene according to their original positions, and the final segmentation mask of the large-scale remote sensing scene with full resolution and spatial consistency is generated by upsampling.

[0109] Furthermore, Step 6 includes:

[0110] Step 6.1, Location Mapping: Based on the patch spatial location recorded during image segmentation, the patch-level probability vector is precisely placed in the corresponding region of the global probability map to form a seamless but low-resolution (patch-level) initial segmentation probability map.

[0111] Step 6.2, Probabilistic Upsampling: Using spatial consistency preservation methods such as bilinear interpolation, the initial probability map is upsampled to restore the full resolution of the original input image. This process fully respects and preserves the scene structural context enhanced by GMM modeling, allowing semantic boundaries to naturally conform to the overall trend of ground feature distribution during the refinement process;

[0112] Step 6.3: Generate the final mask: Perform an argmax operation on the full-resolution probability map for each pixel to determine its final semantic category, thereby generating a spatially coherent and detailed large-scale remote sensing scene semantic segmentation mask. Through this front-end optimization, the final output effectively suppresses fragmentation noise caused by misjudgments of isolated patches and significantly improves the boundary consistency and regional integrity of various land features in complex scenes.

[0113] To verify the effectiveness of this invention in open-vocabulary remote sensing segmentation tasks, extensive evaluations were conducted on 17 open-vocabulary segmentation remote sensing benchmark datasets, following the SegEarth-OV evaluation protocol, covering both multi-class semantic segmentation and single-class extraction tasks. In terms of implementation, this invention employs the MMSegmentation toolchain to implement the algorithm flow. For the visual backbone, the official ViT-B / 16 pre-trained weights from OpenAI are used to load CLIP. For DINOv3, ViT-H+ / 16 is used for the VDD and UAVid datasets, while ViT-L / 16 pre-trained on SAT-493M is used for all other datasets. For the text encoder, standard ImageNet cue templates (e.g., "a photo of a classname") are used. In the single-class extraction task, background cues are further enhanced by combining background cues with multiple fine-grained negative concepts (e.g., forest, vegetation, bare land, river, paved road). For the comparative version, the feedforward neural network and residual connections in the final Transformer layer are removed from MaskCLIP+, and the final self-attention module is replaced with a multi-layer fusion modulation attention mechanism to obtain the comparative MaskCLIP*. This invention was tested for inference on a single-card NVIDIA RTX 4090 platform. The input size used in the experiment was... The system does not use a sliding window at the inference end and triggers an alternating optimization correction process after processing every 50 images, with the number of iterations set to 10 to ensure convergence and efficiency.

[0114] Furthermore, when evaluating the performance of the context-joint reasoning segmentation method of this invention on various datasets, it is divided into multi-class semantic segmentation datasets and single-class object extraction datasets. For multi-class semantic segmentation, this invention evaluated eight datasets: OpenEarthMap, LoveDA, iSAID, Potsdam, Vaihingen2, UAVid, UDD5, and VDD. These datasets contain different land cover categories and different spatial structures. The first five mainly contain high-resolution satellite images, while the latter three focus on UAV urban scene observations. The mean Intersection-over-Union (mIoU) was used as the evaluation metric, and its performance was compared with MaskCLIP, ClearCLIP, GEM, SegEarth-OV, and MaskClip* techniques. The experimental results are shown in Table 1. According to the data, the average mIoU of the present invention reaches 40.78%. Compared with SegEarth-OV, the present invention achieves higher average performance (+2.80%) without relying on any pre-trained upsampler, as shown in Table 1. This indicates that modeling cross-region semantic dependencies at inference time is a more direct and generalizable solution than introducing additional parameter modules.

[0115] Table 1. Quantitative Comparison of Open Vocabulary Remote Sensing Semantic Segmentation Performance

[0116] For uniclass segmentation, ConInfer achieved greater gains in uniclass segmentation than in multiclass segmentation, increasing the baseline average IoU from 28.25% to 39.14%, outperforming SegEarth-OV by an average of 6.13%. (WHU) Sat.II The improvements on Inria, CHN6-CUG, and WBS-SI are particularly significant, as shown in Table 2, demonstrating that context-aware reasoning is highly effective when target categories exhibit strong structural regularity. Buildings and roads typically form spatially continuous, homogeneous patterns, and the contextual prior encoded by DINOv3 allows this invention to reinforce these structured dependencies, resulting in more coherent and complete extraction regions.

[0117] Table 2. Quantitative Comparison of Open Vocabulary Remote Sensing Single-Class Extraction Tasks

[0118] The specific embodiments of the present invention have been described in detail above. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A semantic segmentation method for remote sensing images based on contextual joint reasoning, characterized in that, The method includes: Step 1: Divide the large-scale remote sensing scene into several images of fixed size. Each image patch will be processed independently to extract patch-level features. Step 2: Use the image encoder in the visual language model to divide each image block into patches and extract visual features. Use the text encoder to extract semantic information from the text prompts with embedded categories. Calculate the similarity between the visual features and the text features to obtain the prior probability of the category. Step 3: Extract contextual information from the large visual model. Input each image patch into the basic visual model to extract visual features with robust spatial structure. By fitting a Gaussian mixture model onto the remote sensing scene, the contextual posterior probability distribution of the image is obtained. Step 4: Construct a joint optimization objective function. First, two types of information are introduced as constraints: one is the semantic prior from the visual language model, which provides a category probability distribution based on the text prototype for each image patch; the other is the contextual posterior obtained from the visual base model across the entire scene, which introduces KL divergence to measure the distribution difference and constrains the degree of deviation between the two, thereby organizing semantic information and structural information into a unified optimization objective. Step 5: Alternately optimize the prediction distribution and Gaussian Mixture Model (GMM) parameters using the EM algorithm. First, calculate the context posterior for each patch based on the current structural model. Then, under the condition of fixing the posterior, obtain the calibrated patch-level prediction by fusing the semantic prior and the context posterior. Then, use these calibrated predictions as soft responsibility to update the structural model parameters. Iterate until the target converges, so that visual clustering and language categories are better aligned, and semantic prediction is subject to global space constraints, thereby suppressing isolated or fragmented erroneous predictions. After iterative convergence, output the calibrated semantic probability distribution for each patch. Step 6, Reasoning and Reconstruction: The calibration patch probabilities obtained after optimization and convergence are mapped back to the full scene according to their original positions, and a large-scale remote sensing scene final segmentation mask with full resolution and spatial consistency is generated through upsampling.

2. The remote sensing image semantic segmentation method based on contextual joint reasoning according to claim 1, characterized in that: In Step 1, the large-scale remote sensing scene is divided into several image blocks of fixed pixel size according to the principle of non-overlapping. Each image block is input into the image encoder in the visual base model DINOv3 and the visual language model respectively, and the patch of each image block is processed to extract features.

3. The remote sensing image semantic segmentation method based on contextual joint reasoning according to claim 1, characterized in that: The specific steps of Step 2 include: Step 2.1: For the visual language model branch, input the processed image patches into the image encoder to extract patch-level visual features, and input the category cue template into the text encoder to obtain the category prior probability. Specifically, this includes: first, setting the number of segments and patches; the size of each image patch is W×H, and the pixel size of the patch is p×p; then the number of patches per tile is... Visual encoder using VLM Forward reasoning is performed on each patch to obtain a patch-level visual representation: ； Where n represents the number of patches in each image patch, and d represents the feature dimension; simultaneously, given a set of candidate categories C is a user-defined set of candidate categories. A text prompt is constructed for each category and a text encoder is used. The encoded text prototype vector set T is represented as: ； Step 2.2: Using normalized cosine similarity as a measure of semantic relevance between the patch and the original text, the similarity score is calculated, in the form of: ； in, Category-specific prompt templates, For visual feature vectors, For text feature vectors, It is a temperature hyperparameter; The similarity scores are converted into patch-level class probabilities, scaled using a temperature parameter, and then softmax normalized along the class dimension to obtain the semantic prior distribution, whose components are represented as follows: ； The resulting patch-level semantic prior set is used as the semantic constraint input for the subsequent context joint reasoning module. Together with the context posterior obtained by fitting the visual base model across the entire scene, it participates in joint optimization to achieve semantic and spatial consistency calibration and output the final calibration prediction.

4. The remote sensing image semantic segmentation method based on contextual joint reasoning according to claim 1, characterized in that: The specific steps for extracting contextual information from remote sensing images in Step 3 include: using the visual foundation model DINOv3 to extract high-level visual features from all patches of the entire large-scale scene to form a patch set, capturing the contextual dependencies in the remote sensing image, represented as: To characterize the contextual structure and visual clustering relationships within a scene, a Gaussian Mixture Model (GMM) with k mixture components is fitted onto the feature set X, and its density is expressed as: ； in, Let k be the mixing weights, mean vector, and covariance of the k-th component, respectively. Let the number of mixing components k be the same size as the candidate category set C in the visual language branch, i.e., k = |C|. Based on the current GMM parameters, calculate the contextual posterior probability distribution of each patch feature in each mixing component: 。 5. The remote sensing image semantic segmentation method based on contextual joint reasoning according to claim 1, characterized in that: Step 4 includes: for the i-th patch in the remote sensing scene, introducing a potential consistency probability distribution. As the latent variable to be determined, a prior probability distribution is defined. posterior probability distribution of visual context First, we construct a patch-by-patch consistency minimization objective based on prior and posterior a priori assumptions, formalized as: ； For the potential consensus distribution { Joint learning with GMM parameters allows for contextual posterior learning. As the optimization process gradually incorporates semantic alignment information, the two reinforce each other, and the joint optimization objective is denoted as: 。 6. The remote sensing image semantic segmentation method based on contextual joint reasoning according to claim 1, characterized in that: Step 5 includes employing an alternating optimization strategy based on Expectation-Maximization (EM) to alternately update GMM parameters and patch-level consistency distribution, thereby improving the context posterior. With potential consensus distribution The mutual reinforcement and gradual convergence are achieved through the following specific steps: Step 5.1: Initialize key parameters, starting with the semantic prior distribution from the visual language branch. Initialize the mean and covariance of the Gaussian mixture model, and simultaneously set the mixture weights. Keep it fixed as a uniform distribution Where k is the number of components in the Gaussian mixture model, which is the same as the number of candidate classes, and can be formalized as: ； ； ； in, This represents the probability that the i-th patch belongs to the k-th Gaussian mixture model component; Step 5.2: Alternating Iterative Updates. After the above initialization, execute the following three steps alternately until convergence or the preset maximum number of iterations is reached: (1) E-step: Update the context posterior q, in the current Gaussian mixture model parameters { The responsibility of each patch on each component is calculated posteriorly according to GMM, in the following form: ； (2) Z-update: Update the potential consistency distribution z. Given a fixed semantic prior p and a current context posterior q, obtain the analytical update for each patch according to the KL consistency objective: ； ； Among them, the product operation emphasizes the consistency between 𝑝 and q. 𝑧 is high only when both probabilities are high. If one of them is low, z is suppressed, thus realizing the fusion of prior knowledge and contextual information and enhancing the reliability of understanding. (3) M-step: The updated z is regarded as the soft responsibility of this M-step, and the mean and covariance of the GMM are re-estimated accordingly. The specific update formula is as follows: ； ； Alternatingly executing the above three steps E, Z, and M decreases the objective function, thereby achieving gradual alignment and mutual reinforcement of z and q, ultimately leading to convergence of { } was used as a calibrated patch-level semantic distribution to generate the final segmentation result.

7. The remote sensing image semantic segmentation method based on contextual joint reasoning according to claim 1. Its characteristics are: Step 6 includes: Step 6.1, Location Mapping: Based on the patch spatial location recorded during image segmentation, the patch-level probability vector is precisely placed in the corresponding region of the global probability map to form a seamless but low-resolution initial segmentation probability map. Step 6.2, Probability Upsampling: Using the spatial consistency preservation method of bilinear interpolation, the initial probability map is upsampled to restore the full resolution of the original input image; Step 6.3: Generate the final mask: Perform the argmax operation on the full-resolution probability map for each pixel to determine its final semantic category, thereby generating a spatially coherent and detailed large-scale remote sensing scene semantic segmentation mask. Through this front-end optimization, the final output effectively suppresses fragmentation noise caused by misjudgment of isolated patches and significantly improves the boundary consistency and regional integrity of various land features in complex scenes.

8. A remote sensing image semantic segmentation system based on contextual joint reasoning, characterized in that, The system includes a module for performing the remote sensing semantic segmentation method based on contextual joint reasoning as described in any one of claims 1-7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the remote sensing image semantic segmentation method based on context joint reasoning as described in any one of claims 1-7.

Images