Anti-illusion change detection method for multi-modal large model

Abstract

The invention relates to the technical field of computer vision, in particular to a multi-modal large model illusion change resistance detection method, which comprises the following steps: determining remote sensing images corresponding to a plurality of different time points, calling a preset SigLip image encoder, and extracting corresponding time characteristic differences from the plurality of remote sensing images by using the SigLip image encoder; obtaining text data related to the remote sensing image, inputting the text data and the time characteristic difference into a pre-trained multi-modal large language model for identification, generating a preliminary change detection result, correcting the preliminary change detection result based on a CVA-CICL mechanism, and obtaining a corresponding corrected change detection result; and decomposing the corrected change detection result into a plurality of change detection tasks based on a chain thinking reasoning mechanism, generating a corresponding candidate solution for each change detection task, aggregating all the candidate solutions, and outputting a corresponding final change detection result. According to the invention, the adaptability and accuracy of change detection can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision, and in particular to a multimodal large model anti-hallucination change detection method. Background Art

[0002] Change Detection (CD) is a critical task in remote sensing that aims to identify and analyze temporal changes in natural hazard assessment and environmental conditions. Accurate change detection methods are crucial for multiple application areas, including landslide mapping, urban planning, hazard assessment, ecological monitoring, and land management. With the rapid development of satellite and aerial image acquisition technologies, there is an increasing demand for automated change detection tools that can handle large-scale datasets and extract meaningful insights from high-resolution imagery. Traditional methods, such as image difference and post-classification comparison, have limited effectiveness due to their inability to handle complex and dynamic environments. Deep Learning (DL) methods have therefore been widely used in change detection tasks, especially in the use of Convolutional Neural Networks (CNN) and Transformer architectures, due to their ability to learn complex spatiotemporal patterns.

[0003] Existing change detection methods are widely used in remote sensing image analysis, but they still face some problems and challenges, especially when dealing with change detection in complex and dynamic environments. Traditional change detection methods (such as image difference method and post-classification method) are easily limited by environmental noise, subtle changes and large-scale data set processing when faced with large amounts of spatiotemporal data and high-resolution images, and are difficult to effectively capture complex change patterns. In addition, although convolutional neural networks (CNNs) and transformers based on deep learning have improved the accuracy of change detection to a certain extent, they still have deficiencies in local feature extraction and generalization capabilities outside the distribution of training data. Moreover, these methods are prone to hallucinations, which affects the reliability of the results.

[0004] In order to solve the above problems, the present invention proposes a change detection method based on a multimodal large language model, aiming to improve the adaptability and accuracy of change detection by introducing an innovative framework. Summary of the invention

[0005] In order to improve the adaptability and accuracy of change detection, the present application provides a multimodal large model anti-hallucination change detection method.

[0006] In a first aspect, the present application provides a multimodal large model anti-hallucination change detection method, which adopts the following technical solution:

[0007] A multimodal large model anti-hallucination change detection method comprises: determining remote sensing images corresponding to multiple different time points, calling a pre-set SigLip image encoder, and using the SigLip image encoder to extract corresponding time feature differences from the multiple remote sensing images, wherein the remote sensing images include spatial information and texture features; obtaining text data related to the remote sensing images, inputting the text data and the time feature differences into a pre-trained multimodal large language model for recognition, and generating a preliminary change detection result; correcting the preliminary change detection result based on a CVA-CICL mechanism to obtain a corresponding corrected change detection result, wherein the preliminary change detection result includes an identification of a change area and its possible change type; decomposing the corrected change detection result into multiple change detection tasks based on a chain thinking reasoning mechanism, and generating a corresponding candidate solution for each of the change detection tasks, aggregating all the candidate solutions, and outputting a corresponding final change detection result.

[0008] Optionally, in the process of obtaining the corresponding revised change detection result, the method also includes: retrieving the time feature difference, and obtaining the pixel-level information between multiple remote sensing images based on the time feature difference; retrieving the preliminary change detection result, and iteratively calibrating the pixel-level information and the preliminary change detection result through the CVA-CICL mechanism to obtain the corresponding revised change detection result.

[0009] Optionally, the SigLip image encoder includes a convolutional neural network and a transformer architecture, and the method also includes: using a multi-layer filter of a convolutional neural network to scan and extract local features of multiple remote sensing images layer by layer, transferring local features through an activation function to construct a high-dimensional feature map, the local features include edges and textures, the encoder uses direct subtraction of the high-dimensional feature map, absolute difference calculation or complex measurement methods to generate a feature vector representing time changes, the encoder integrates the feature vectors to determine the corresponding time feature differences; and / or, using a transformer architecture, multiple remote sensing images are divided into multiple patches and flattened into one-dimensional vectors, and a multi-head attention mechanism is used to enable each pixel to pay attention to other important areas on the entire picture, thereby capturing global context information, and the feedforward neural network inside each Transformer block transforms the feature space, and the feature representation of local details and global context information is fused through multiple layers of Transformer blocks to determine the time feature difference.

[0010] Optionally, in the process of generating preliminary change detection results, the method also includes: obtaining text data related to the remote sensing image, and using a pre-trained language model to parse semantic information in the text data, wherein the text data includes descriptions of image changes, geographic location, timestamps, user annotations, historical records or other contextual information; fusing the semantic information with image features to generate a comprehensive representation, wherein the comprehensive representation includes spatial features of the image and additional background knowledge provided by the text; based on the comprehensive representation, identifying and classifying the change areas that have occurred, and outputting preliminary change detection results based on the change areas.

[0011] Optionally, the method also includes: performing pixel-level change detection on the preliminary change detection results using the CVA function in each iteration, and performing multiple rounds of change detection to evaluate the coordinate position corresponding to the change area detected initially; identifying inaccurate boundaries and making corrections based on the coordinate position of the change area by comparing the changes in adjacent pixels; in the correction process, screening and excluding areas that are not related to actual changes or are incorrectly marked as change areas, and generating adjusted corrected change detection results after multiple rounds of iterative calibration.

[0012] Optionally, in the process of outputting the corresponding final change detection result, the method also includes: decomposing the change detection task into multiple subtasks, wherein each subtask is used to solve the change detection problem of a specific type or area; generating multiple candidate solutions based on each subtask, wherein the candidate solutions are reasoning paths, performing self-consistency checks and screening on the generated multiple reasoning paths, performing information aggregation on the reasoning paths that have been screened and retained, and outputting the final change detection result after processing by the chain thinking reasoning mechanism.

[0013] In a second aspect, the present application provides a multimodal large model anti-hallucination change detection method, which adopts the following technical solution:

[0014] A multimodal large model anti-hallucination change detection device, comprising:

[0015] A time feature difference extraction module determines remote sensing images corresponding to multiple different time points, calls a preset SigLip image encoder, and uses the SigLip image encoder to extract corresponding time feature differences from multiple remote sensing images, wherein the remote sensing images include spatial information and texture features;

[0016] A modified change detection result acquisition module is used to acquire text data related to the remote sensing image, input the text data and the time feature difference into a pre-trained multimodal large language model for recognition, and generate a preliminary change detection result. The preliminary change detection result is modified based on the CVA-CICL mechanism to obtain a corresponding modified change detection result, wherein the preliminary change detection result includes an identification of a changed area and its possible change type;

[0017] The final change detection result output module decomposes the modified change detection result into multiple change detection tasks based on the chain thinking reasoning mechanism, generates a corresponding candidate solution for each change detection task, aggregates all the candidate solutions, and outputs the corresponding final change detection result.

[0018] Optionally, the device further comprises:

[0019] A pixel-level information acquisition module is used to retrieve time feature differences and acquire pixel-level information between multiple remote sensing images based on the time feature differences;

[0020] The iterative calibration module retrieves the preliminary change detection result and is used to iteratively calibrate the pixel-level information with the preliminary change detection result through the CVA-CICL mechanism to obtain the corresponding corrected change detection result.

[0021] In a third aspect, the present application provides a multimodal large model anti-hallucination change detection method, which adopts the following technical solution:

[0022] A multimodal large model anti-hallucination change detection method comprises a processor, wherein the processor runs a program of any one of the multimodal large model anti-hallucination change detection methods described above.

[0023] In a fourth aspect, the present application provides a storage medium, which adopts the following technical solution:

[0024] A storage medium stores a program of any one of the multimodal large model anti-hallucination change detection methods described above.

[0025] In summary, the present application includes at least one of the following beneficial technical effects:

[0026] 1. By deeply integrating the temporal feature differences of remote sensing images with relevant text data, the ability to understand complex environmental changes has been significantly improved. The pre-trained language model is used to parse the semantic information in the text and comprehensively represent it with the image features, so that the model can not only capture visual changes, but also make more accurate and comprehensive judgments based on contextual knowledge. This efficient integration of multimodal information ensures higher adaptability when processing different types of remote sensing images.

[0027] 2. The CVA-CICL mechanism introduces an iterative adjustment process at the pixel level, effectively filtering out irrelevant or false detection results, greatly improving the spatial accuracy of change detection. By repeatedly evaluating and correcting the coordinate position and boundaries of the changed area, the mechanism can reduce false positives while maintaining details, thereby providing more reliable change detection output. This process is especially important for small changes, ensuring accurate results even in the presence of more noise or poor image quality.

[0028] 3. The chained reasoning mechanism decomposes the change detection task into multiple subtasks and generates multiple reasoning paths to explore different change patterns. By performing strict self-consistency checks and information aggregation on these paths, the mechanism not only enhances the logical coherence and reliability of the detection results, but also performs well in the face of complex and dynamic environments. This approach ensures that the final output not only considers multiple possibilities, but also selects the most reliable results, greatly improving the overall robustness and accuracy of change detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 The present invention is a flowchart of a multimodal large model anti-hallucination change detection method according to an exemplary embodiment.

[0030] Figure 2 It is a structural block diagram of a multimodal large model anti-hallucination change detection device according to an exemplary embodiment. DETAILED DESCRIPTION

[0031] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings.

[0032] In the description of this specification, the description with reference to the terms "certain embodiments", "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0033] The present application embodiment discloses a multimodal large model anti-hallucination change detection method, referring to Figure 1 ,include:

[0034] S100, determining a plurality of remote sensing images corresponding to different time points, calling a preset SigLip image encoder, and using the SigLip image encoder to extract corresponding time feature differences from the plurality of remote sensing images.

[0035] Among them, remote sensing images include spatial information and texture features. Multiple remote sensing images from different time points are used as input. These images may have different spatial resolutions, spectral bands, field of view angles and other characteristics. It should be pointed out here that remote sensing images need to be preprocessed, including denoising, alignment and normalization, to ensure the consistency and accuracy of subsequent analysis.

[0036] For feature extraction, the SigLip image encoder includes a convolutional neural network and a transformer architecture. The method specifically includes:

[0037] S101 uses a convolutional neural network's multi-layer filter to scan and extract local features of multiple remote sensing images layer by layer, and transfers local features through activation functions to construct a high-dimensional feature map. Local features include edges and textures. The encoder uses high-dimensional feature maps to directly subtract, calculate absolute differences, or use complex metrics to generate feature vectors that represent time changes. The encoder integrates the feature vectors to determine the corresponding time feature differences.

[0038] Taking two remote sensing images as an example, the CNN layer in the SigLip image encoder uses multiple layers of filters to scan each remote sensing image layer by layer, and each filter is responsible for detecting a specific type of local features, such as edges, textures and other morphological features. The output after convolution is processed by an activation function (such as ReLU) to introduce nonlinear characteristics. The role of the activation function is to enable the model to learn more complex patterns and pass the processed features to the next layer.

[0039] Through the above steps, CNN generates a set of high-dimensional feature maps, which represent the information of the original image at different levels of abstraction and capture rich local spatial features. They can be directly subtracted, calculated by absolute difference, or by complex metrics.

[0040] Direct subtraction or absolute difference calculation: For two remote sensing images at different time points, their corresponding high-dimensional feature maps can be directly subtracted, or the absolute difference can be calculated to quantify the changes between the two.

[0041] Complex metrics: In addition to simple subtraction or absolute difference, more complex metrics (such as Euclidean distance, cosine similarity, etc.) can be used to capture spatial and temporal differences more accurately.

[0042] The temporal feature differences generated by the above method are organized into a feature vector representing temporal changes, and the feature vector not only reflects the temporal and spatial changes between images, but also retains important structural information; in order to further reduce the feature dimension and retain key structural information, pooling layers can be introduced. Common pooling operations include max pooling and average pooling, and batch normalization is added to stabilize the training process and accelerate convergence, preventing overfitting while improving model performance; finally, the temporal feature differences extracted from CNN are organized into compact and expressive feature embeddings, which are then passed as input to the large language model for subsequent change detection tasks.

[0043] The local features in remote sensing images are efficiently extracted through convolutional neural networks (CNN), and the temporal differences are quantified in a variety of ways to generate accurate temporal feature difference embeddings, providing high-quality basic data for subsequent change detection.

[0044] S102, and / or, using a transformer architecture, multiple remote sensing images are divided into multiple patches and flattened into a one-dimensional vector, and a multi-head attention mechanism is used to enable each pixel to focus on other important areas on the entire image, thereby capturing global context information, and the feedforward neural network inside each Transformer block transforms the feature space, and the feature representation of local details and global context information is fused through multiple layers of Transformer blocks to determine the temporal feature difference.

[0045] Similarly, the input is multiple remote sensing images from different time points. Since Transformer does not require a fixed grid input format, it can flexibly process data of any shape. Therefore, each remote sensing image is divided into several patches of fixed size. This is done so that each patch can be processed independently while maintaining the spatial structure of the image; and each patch is flattened into a one-dimensional vector for subsequent processing. This step allows each patch to be regarded as an element in the sequence.

[0046] In order to preserve the spatial relationship, a position encoding is added to each patch. The position encoding provides information about the relative position of the patch in the original image, which is crucial for understanding the overall structure of the image.

[0047] The core of Transformer lies in its self-attention mechanism, in which each pixel can not only focus on itself, but also consider other important areas of the entire image. Through the multi-head attention mechanism, the model can analyze data from multiple angles at the same time and enhance the ability to understand complex patterns.

[0048] It should be pointed out here that each "head" works independently, but the final results will be merged to form a comprehensive feature representation. This mechanism allows the model to take into account the information of the entire sequence when processing each position, thereby better understanding the changing patterns in complex environments.

[0049] Each Transformer block also includes a feedforward neural network consisting of two fully connected layers, which is used to further transform the feature space and increase the nonlinear expression ability of the model. The role of FFN is to adjust the feature representation to make it more suitable for capturing complex spatiotemporal changes. After being processed by multiple layers of Transformer blocks, the feature representation that combines local details and global context information is organized into feature embeddings suitable for large language models for the next change detection step.

[0050] By capturing the global contextual information in remote sensing images through the transformer architecture and combining it with a multi-head attention mechanism and a feedforward neural network, the ability to understand complex environmental changes is significantly enhanced, especially for the recognition of subtle and dynamic changes with higher accuracy and robustness.

[0051] S110, obtaining text data related to the remote sensing image, inputting the text data and the time feature difference into a pre-trained multimodal large language model for recognition, generating a preliminary change detection result, correcting the preliminary change detection result based on the CVA-CICL mechanism, and obtaining a corresponding corrected change detection result.

[0052] In the process of generating the preliminary change detection result, specifically, the method further includes:

[0053] S111, obtaining text data related to the remote sensing image, and using a pre-trained language model to parse semantic information in the text data.

[0054] Among them, text information related to remote sensing images is obtained from multiple sources, which may include but are not limited to descriptions of image changes, geographic locations (such as coordinates or area names), timestamps (recording the time when the image was captured), user annotations (such as expert annotations or comments), historical records (previous changes in the same location), and any other available contextual information. Text data can come from databases, metadata tags, online platforms, or documents provided by domain experts.

[0055] At the same time, the collected text is cleaned and standardized to remove unnecessary characters, format the text content to ensure consistency, and convert the unstructured text into a form suitable for analysis. This step may involve natural language processing techniques such as word segmentation, stop word removal, and stemming, with the aim of improving the quality of subsequent semantic analysis.

[0056] First, select a pre-trained language model (LLM) suitable for processing text in a specific field, such as BERT, RoBERTa, T5 or other models designed for understanding text related to remote sensing images. These models have been trained on large-scale corpora and can understand complex sentence structures and vocabulary meanings, which helps to parse professional terms and technical descriptions related to remote sensing images. Then, the pre-processed text is input into the selected language model, and each word or phrase is mapped to a high-dimensional vector space through an embedding layer. This process adjusts the representation of words according to the context so that words with similar meanings are close to each other in the vector space.

[0057] The language model uses its internal architecture (such as multi-layer bidirectional encoders) to capture long-distance dependencies and deep semantic information in the text. It not only focuses on the meaning of individual words, but also considers the logical coherence of the entire sentence or even the paragraph, thereby parsing the complete intention and background knowledge behind the text. Ultimately, the language model outputs a series of vectors rich in semantic information as representations of text data, which contain an understanding of the description of changes in remote sensing images, an explanation of geographic location, time series information, and any other relevant details that are helpful for change detection.

[0058] S112, the semantic information is integrated with the image features to generate a comprehensive representation.

[0059] Among them, before this step, the SigLip image encoder has completed the extraction of the temporal feature differences of two remote sensing images at different time points and generated corresponding feature vectors. These feature vectors represent the spatial features of the image, such as local features such as edges and textures, as well as the overall spatial layout.

[0060] You can use a simple concatenation method to directly connect the two vectors, or you can use a more complex strategy such as the attention mechanism to let the model automatically learn how to weight the information between the two; the attention mechanism allows the model to focus on those parts that are most critical to the current task, such as certain specific types of changes or changes in a specific area, rather than just evenly mixing all the information.

[0061] It should be pointed out here that the comprehensive representation integrates complementary information from images and texts, retaining the spatial features carried by the image itself and adding additional background knowledge provided by the text, such as the specific nature of the change, the geographical location where it occurred, and its time frame. This comprehensive representation enables the model to understand environmental changes more comprehensively and improves the accuracy of change detection.

[0062] S113, identifying and classifying the changed areas based on the comprehensive representation, and outputting preliminary change detection results based on the changed areas.

[0063] Among them, the comprehensive representation is used as input to identify the changed areas in the image through machine learning algorithms (such as support vector machines, random forests) or deep learning networks (such as convolutional neural networks CNN). The model will determine which pixels have changed significantly based on the information in the comprehensive representation and mark the locations of these pixels.

[0064] Furthermore, based on the rich information in the comprehensive representation, the identified change areas can be classified to determine which type of change they belong to, such as building construction, vegetation cover change, water area expansion, etc. The classifier can automatically infer the category of change based on the features in the comprehensive representation, providing more detailed change detection results.

[0065] Finally, the system generates a preliminary change detection report based on the results of recognition and classification. This report usually contains the identification of the changed areas and their possible change types, and sometimes also includes a confidence score to help users evaluate the reliability of the detection results.

[0066] By combining the semantic information of text data parsed by the pre-trained language model with image features, the generated comprehensive representation not only enhances the model's ability to understand complex environmental changes, but also improves the accuracy and robustness of change detection.

[0067] In the process of obtaining the corresponding modified change detection result, the method further includes:

[0068] S114, retrieving time feature differences, and acquiring pixel-level information between multiple remote sensing images based on the time feature differences.

[0069] The specific implementation process includes the following steps:

[0070] 1. Calling time feature difference data: The system first calls the previously generated time feature differences from the SigLip image encoder. These time feature differences are obtained by comparing remote sensing images at different time points, and they contain rich temporal and spatial change information.

[0071] 2. Extract pixel-level information: For each pair of remote sensing images at a time point, the system will further refine the temporal feature differences to the pixel level. This includes but is not limited to: Spatial resolution matching: ensuring that all images involved in the comparison have the same spatial resolution so that each pixel can be directly compared; Pixel value quantification: converting the numerical value in the temporal feature difference into a specific pixel value change, such as changes in brightness, color or texture; Multi-dimensional feature analysis: In addition to simple brightness changes, other factors that may affect change detection are also considered, such as edge sharpness, texture complexity, etc.

[0072] Through the above steps, one or more high-resolution difference maps are generated, in which each pixel reflects the degree of change of the corresponding position between two time points. These difference maps not only provide accurate spatial positioning, but also retain rich detail information, which is helpful for subsequent fine calibration.

[0073] S115, retrieve the preliminary change detection result, iteratively calibrate the pixel-level information and the preliminary change detection result through the CVA-CICL mechanism, and obtain the corresponding revised change detection result.

[0074] The specific execution process of iterative calibration includes the following steps:

[0075] The system retrieves the preliminary generated change detection results from the multimodal large language model (LLM), which usually contains the identification of the changed area and its possible change type, but there may be some false positives or omissions. The high-resolution difference map generated above is combined with the preliminary change detection results as the input of the CVA-CICL mechanism. This combination allows the system to use more detailed pixel-level information for precise correction based on existing predictions.

[0076] The CVA-CICL mechanism starts an iterative loop that aims to gradually optimize the change detection output. Each iteration uses the Change Vector Analysis (CVA) function to perform pixel-level change detection and adjust the previous predictions based on the new findings. In each iteration, the CVA-CICL mechanism re-evaluates the coordinate position of the initially detected change area. It not only considers the results of the current round, but also refers to the detection results of the previous rounds to ensure the accuracy of the change coordinates. This multi-round feedback mechanism can effectively reduce false positives caused by noise or other factors. The boundaries of the change area are refined, and inaccurate boundaries are identified and corrected by comparing the changes in adjacent pixels; this process ensures that the final output change area is not only accurate in position, but also has clear boundaries.

[0077] Throughout the iterative process, the CVA-CICL mechanism continuously screens and excludes areas that are irrelevant to actual changes or are incorrectly marked as changes; this step is critical to maintaining the quality of the final output because it is directly related to the reliability and credibility of change detection. After multiple rounds of iterative calibration, the CVA-CICL mechanism generates finely tuned change detection results; each round of iteration will improve on the results of the previous round until the preset convergence conditions are reached or specific accuracy requirements are met.

[0078] Finally, the system outputs the corrected change detection results after iterative calibration using the CVA-CICL mechanism. These results are not only greatly improved in accuracy, but can also effectively handle small changes, making them suitable for high-precision application scenarios such as land management, urban planning, and environmental monitoring. Through iterative calibration using the CVA-CICL mechanism, the system can make precise adjustments to the preliminary change detection results based on pixel-level information, significantly improving the accuracy of change detection.

[0079] S120, based on the chain thinking reasoning mechanism, the modified change detection result is decomposed into multiple change detection tasks, and a corresponding candidate solution is generated for each change detection task, all the candidate solutions are aggregated, and the corresponding final change detection result is output.

[0080] In the process of outputting the corresponding final change detection result, specifically, the method further includes:

[0081] S121, decomposing the change detection task into multiple subtasks, wherein each subtask is used to solve a change detection problem of a specific type or region;

[0082] Among them, the change detection task is decomposed into the execution process of multiple subtasks:

[0083] First, the system analyzes the preliminary and revised change detection results to identify different types of change patterns, which may include, for example, the construction and demolition of buildings, changes in vegetation cover, increases or decreases in water areas, etc. The entire detection area is divided into several sub-areas based on the spatial distribution of remote sensing images. Each sub-area can be a specific urban block, a forest, a lake or river, etc., ensuring that each subtask focuses on solving a specific type of local change.

[0084] Based on the above analysis, the system generates a series of subtasks, each of which is responsible for handling a specific type of change or a specific geographical area. This subdivision helps to concentrate resources to solve specific problems and improve detection accuracy. For some complex scenarios, there may be multiple types of changes that are interrelated. In this case, the system will set the priority and dependency between subtasks to ensure that they are processed in a reasonable order.

[0085] S122, generate multiple candidate solutions based on each subtask, perform self-consistency check and screening on the generated multiple reasoning paths, aggregate information on the reasoning paths that have been screened and retained, and output the final change detection result after processing by the chain thinking reasoning mechanism.

[0086] For each subtask, the system proposes multiple hypotheses from different perspectives to explain possible changes. For example, for a suspected change area, there may be different explanations based on multiple features such as shape, color, and texture. Using pre-trained deep learning models or other machine learning algorithms, each hypothesis is predicted to generate multiple candidate solutions. Each solution is a reasoning path that represents a possible change pattern.

[0087] To ensure the diversity of reasoning paths, the system can introduce certain random perturbations in the input data, or use different initialization parameters within the model to explore a wider range of possibilities; combine historical records and other contextual information, such as past land use or seasonal changes, to further enrich the range of choice for reasoning paths.

[0088] The specific execution process of self-consistency checking and screening the reasoning path includes:

[0089] Each reasoning path undergoes a rigorous logical consistency check to ensure that it is internally logically coherent and free of contradictions. For example, if one path suggests that some buildings have been demolished while another path finds new buildings at the same location, the two paths need to be re-evaluated.

[0090] In addition to the internal logic, the system also checks whether each path has sufficient external evidence support, which includes but is not limited to other sensor data, literature, expert opinions, etc., to enhance the credibility of the path.

[0091] The system evaluates the mutual support between different reasoning paths and retains those paths that can form good connections with other paths and have internal logical coherence. For example, if two paths explain the same change phenomenon from different perspectives and complement each other, they are more likely to be correct. Based on the above evaluation results, the system scores each reasoning path, taking into account factors such as logical consistency, external evidence support, and the degree of mutual support between paths; finally, the paths with the highest scores are selected as candidate solutions.

[0092] The execution process of information aggregation and output of final change detection results:

[0093] For the reasoning paths that have been screened and retained, the system uses a weighted average method to merge their results. The weights can be dynamically adjusted according to factors such as the credibility and strength of evidence of each path to ensure that the final output is more reliable. Another common method is to use a majority voting mechanism to select the result that most paths point to as the final conclusion. This method is particularly suitable for discrete change classification problems.

[0094] When outputting the final change detection results, the system will also provide a confidence score to reflect the reliability of the result. A high confidence score indicates that the result is relatively certain, while a low confidence score prompts the user to further verify. The final change detection results can be intuitively displayed through visualization tools, such as marking the change area and its type on the map. In addition, the system can also generate a detailed detection report, including text descriptions, statistical analysis, etc., to help users better understand and apply the detection results.

[0095] Case background setting

[0096] To monitor significant changes in urban areas from 2019 to 2024, including building construction and demolition, changes in vegetation cover, and increases and decreases in water areas, we applied a multimodal large-model anti-hallucination change detection method. This process involves not only the processing of image data, but also the parsing and fusion of text data, ensuring a comprehensive understanding of complex environmental changes.

[0097] First, high-quality remote sensing images corresponding to multiple different time points are determined, and the pre-set SigLip image encoder is called to extract the temporal feature differences of these images. The acquired remote sensing images cover different spatial resolutions, spectral bands, and field of view, so preprocessing such as denoising, alignment, and normalization is required to ensure the consistency and accuracy of subsequent analysis. Through these preprocessing steps, the comparability between images at different time points is guaranteed, laying a solid foundation for subsequent feature extraction.

[0098] In the feature extraction stage, convolutional neural networks (CNNs) and transformer architectures are used to efficiently extract local features and global context information from remote sensing images. CNNs scan each image layer by layer through multi-layer filters to capture morphological features such as edges and textures, and construct high-dimensional feature maps through activation functions; while transformers capture important areas on the entire image through a multi-head attention mechanism, integrating local details and global context information. The generated temporal feature differences are quantized into feature vectors representing temporal changes, and are further optimized through pooling layers and batch normalization, ultimately forming compact and expressive feature embeddings, providing high-quality basic data for subsequent change detection.

[0099] At the same time, text data related to remote sensing images are collected. These data come from a wide range of sources, including image change descriptions, geographic location information, timestamps, user annotations, historical records, etc. After preprocessing such as cleaning, word segmentation, and removal of stop words, the text data is parsed using a pre-trained language model to analyze its semantic information. The language model can capture long-distance dependencies and deep semantics in the text and output a vector representation rich in semantic information. This semantic information is fused with image features to generate a comprehensive representation that integrates complementary information from images and texts, enhancing the ability to understand environmental changes.

[0100] Based on the comprehensive characterization, the system identifies and classifies the changed areas and outputs preliminary change detection results. This result includes the identification of the changed areas and their possible change types, with confidence scores to help evaluate the reliability of the detection results. Subsequently, the pixel-level information in the temporal feature differences is retrieved and combined with the preliminary change detection results, and iterative calibration is performed through the CVA-CICL mechanism. This mechanism continuously refines the accuracy of the change coordinates, refines the boundaries of the changed areas, and screens and excludes irrelevant or incorrectly marked changed areas, thereby improving the quality of the final output.

[0101] Finally, based on the chain thinking reasoning mechanism, the revised change detection results are decomposed into multiple subtasks to generate candidate solutions for each specific type of change or geographical area. By performing self-consistency checks, external evidence support assessments, and mutual support evaluations on these paths, the system selects the best solution and aggregates all candidate solutions to output the final change detection results. This result not only improves the accuracy of change detection, but also can effectively handle minor changes, and is suitable for application scenarios with high precision requirements such as land management, urban planning, and environmental monitoring.

[0102] Through the above detailed data processing and analysis process, accurate monitoring of land use changes in urban areas over a five-year period was achieved. This method not only improves the accuracy and robustness of change detection, but also provides reliable data support for decision makers, which helps to make more scientific urban management and environmental protection decisions.

[0103] The present application embodiment discloses a multi-modal large model anti-hallucination change detection device, referring to Figure 2 , the devices include but are not limited to:

[0104] The time feature difference extraction module 200 determines remote sensing images corresponding to multiple different time points, calls a preset SigLip image encoder, and uses the SigLip image encoder to extract corresponding time feature differences from the multiple remote sensing images, wherein the remote sensing images include spatial information and texture features;

[0105] The modified change detection result acquisition module 210 acquires text data related to the remote sensing image, inputs the text data and the time feature difference into a pre-trained multimodal large language model for recognition, generates a preliminary change detection result, and modifies the preliminary change detection result based on the CVA-CICL mechanism to obtain a corresponding modified change detection result, wherein the preliminary change detection result includes an identification of the changed area and its possible change type;

[0106] The final change detection result output module 220 decomposes the modified change detection result into multiple change detection tasks based on the chain thinking reasoning mechanism, generates a corresponding candidate solution for each change detection task, aggregates all the candidate solutions, and outputs the corresponding final change detection result.

[0107] Furthermore, the device includes but is not limited to:

[0108] A pixel-level information acquisition module retrieves time feature differences and is used to obtain pixel-level information between multiple remote sensing images based on the time feature differences;

[0109] The iterative calibration module retrieves the preliminary change detection result and is used to iteratively calibrate the pixel-level information with the preliminary change detection result through the CVA-CICL mechanism to obtain the corresponding corrected change detection result.

[0110] Furthermore, the device includes but is not limited to:

[0111] A convolutional neural network module is used to use the multi-layer filter of the convolutional neural network to scan and extract the local features of multiple remote sensing images layer by layer, and to transfer the local features through the activation function to construct a high-dimensional feature map. The local features include edges and textures. The encoder uses direct subtraction of the high-dimensional feature map, absolute difference calculation or complex measurement methods to generate a feature vector representing time changes. The encoder integrates the feature vectors to determine the corresponding time feature differences;

[0112] And / or, a transformer architecture module, for using a transformer architecture, dividing multiple remote sensing images into multiple patches and flattening them into a one-dimensional vector, and utilizing a multi-head attention mechanism to enable each pixel to focus on other important areas of the entire image, thereby capturing global context information, and the feedforward neural network within each Transformer block transforms the feature space, and the feature representation of local details and global context information is fused through multiple layers of Transformer blocks to determine the temporal feature difference.

[0113] Furthermore, the device includes but is not limited to:

[0114] A semantic information parsing module obtains text data related to remote sensing images and uses a pre-trained language model to parse the semantic information in the text data, where the text data includes descriptions of image changes, geographic locations, timestamps, user comments, historical records, or other contextual information;

[0115] A comprehensive representation generation module, which fuses semantic information with image features to generate a comprehensive representation, where the comprehensive representation includes the spatial features of the image and additional background knowledge provided by the text;

[0116] The change region identification and classification module is used to identify and classify the change regions based on the comprehensive representation, and outputs preliminary change detection results based on the change regions.

[0117] Furthermore, the device includes but is not limited to:

[0118] The coordinate position evaluation module uses the CVA function to perform pixel-level change detection on the preliminary change detection results in each iteration, and performs multiple rounds of change detection to evaluate the coordinate position corresponding to the initially detected change area;

[0119] An adjacent pixel comparison module is used to identify inaccurate boundaries and make corrections by comparing changes in adjacent pixels based on the coordinate position of the changed area;

[0120] The screening module, during the correction process, is used to screen and exclude areas that are irrelevant to the actual changes or are incorrectly marked as changed areas. After multiple rounds of iterative calibration, an adjusted and corrected change detection result is generated.

[0121] Furthermore, the device includes but is not limited to:

[0122] A task decomposition module is used to decompose the change detection task into multiple subtasks, wherein each subtask is used to solve a change detection problem of a specific type or area;

[0123] The candidate solution generation module is used to generate multiple candidate solutions based on each subtask, where the candidate solution is the reasoning path. The multiple generated reasoning paths are self-consistency checked and screened, and information is aggregated for the reasoning paths that have been screened and retained, and the final change detection results processed by the chain thinking reasoning mechanism are output.

[0124] An embodiment of the present application further discloses a multimodal large model anti-hallucination change detection method, comprising a processor, wherein a program of any one of the multimodal large model anti-hallucination change detection methods described above is run in the processor.

[0125] The embodiment of the present application also discloses a storage medium storing a program of the multimodal large model anti-hallucination change detection method described in any one of the above.

[0126] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A multimodal large model anti-hallucination change detection method, characterized in that: include: Determine remote sensing images corresponding to multiple different time points, call a preset SigLip image encoder, and use the SigLip image encoder to extract corresponding time feature differences from the multiple remote sensing images, wherein the remote sensing images include spatial information and texture features; Acquire text data related to the remote sensing image, input the text data and the time feature difference into a pre-trained multimodal large language model for recognition, generate a preliminary change detection result, modify the preliminary change detection result based on the CVA-CICL mechanism, and obtain a corresponding modified change detection result, wherein the preliminary change detection result includes an identification of the changed area and its possible change type; Based on the chain thinking reasoning mechanism, the modified change detection result is decomposed into multiple change detection tasks, and a corresponding candidate solution is generated for each change detection task. All candidate solutions are aggregated to output the corresponding final change detection result.

2. The multimodal large model anti-hallucination change detection method according to claim 1, characterized in that: In the process of obtaining the corresponding modified change detection result, the method further includes: Retrieving time feature differences, and obtaining pixel-level information between multiple remote sensing images based on the time feature differences; The preliminary change detection result is retrieved, and the pixel-level information and the preliminary change detection result are iteratively calibrated through the CVA-CICL mechanism to obtain a corresponding corrected change detection result.

3. The multimodal large model anti-hallucination change detection method according to claim 1, characterized in that: The SigLip image encoder includes a convolutional neural network and a transformer architecture, and the method further includes: Use the multi-layer filter of the convolutional neural network to scan and extract the local features of multiple remote sensing images layer by layer, and transfer the local features through the activation function to construct a high-dimensional feature map. The local features include edges and textures. The encoder uses direct subtraction, absolute difference calculation or complex measurement methods of the high-dimensional feature map to generate a feature vector representing the time change. The encoder integrates the feature vectors to determine the corresponding time feature differences; And / or, using a transformer architecture, multiple remote sensing images are divided into multiple patches and flattened into a one-dimensional vector, and a multi-head attention mechanism is used to enable each pixel to focus on other important areas on the entire image, thereby capturing global context information, and the feedforward neural network inside each Transformer block transforms the feature space, and the feature representation of local details and global context information is fused through multiple layers of Transformerblocks to determine the temporal feature difference.

4. The multimodal large model anti-hallucination change detection method according to claim 1, characterized in that: In the process of generating preliminary change detection results, the method further includes: Acquire text data related to the remote sensing image and parse semantic information in the text data using a pre-trained language model, wherein the text data includes descriptions of image changes, geographic locations, timestamps, user annotations, historical records, or other contextual information; The semantic information is integrated with the image features to generate a comprehensive representation, where the comprehensive representation includes the spatial features of the image and the additional background knowledge provided by the text; Based on the comprehensive representation, the changed regions are identified and classified, and preliminary change detection results are output based on the changed regions.

5. The multimodal large model anti-hallucination change detection method according to claim 2, characterized in that: In the process of iteratively calibrating the pixel level information and the preliminary change detection result through the CVA-CICL mechanism, the method further includes: In each iteration, the CVA function is used to perform pixel-level change detection on the preliminary change detection results, and multiple rounds of change detection are performed to evaluate the coordinate positions corresponding to the preliminary detected change areas; Based on the coordinate position of the changed area, the inaccurate boundary is identified and corrected by comparing the changes of adjacent pixels; During the correction process, areas that are irrelevant to the actual changes or are incorrectly marked as changed areas are screened and excluded. After multiple rounds of iterative calibration, adjusted and corrected change detection results are generated.

6. The multimodal large model anti-hallucination change detection method according to claim 1, characterized in that: In the process of outputting the corresponding final change detection result, the method further includes: Decompose the change detection task into multiple subtasks, where each subtask is used to solve the change detection problem of a specific type or area; Based on each subtask, multiple candidate solutions are generated, where the candidate solutions are reasoning paths. The generated multiple reasoning paths are self-consistency checked and screened, and information aggregation is performed on the screened and retained reasoning paths to output the final change detection results after processing by the chain thinking reasoning mechanism.

7. A multi-modal large model anti-hallucination change detection device, characterized in that: include: A time feature difference extraction module determines remote sensing images corresponding to multiple different time points, calls a preset SigLip image encoder, and uses the SigLip image encoder to extract corresponding time feature differences from multiple remote sensing images, wherein the remote sensing images include spatial information and texture features; A modified change detection result acquisition module is used to acquire text data related to the remote sensing image, input the text data and the time feature difference into a pre-trained multimodal large language model for recognition, and generate a preliminary change detection result. The preliminary change detection result is modified based on the CVA-CICL mechanism to obtain a corresponding modified change detection result, wherein the preliminary change detection result includes an identification of a changed area and its possible change type; The final change detection result output module decomposes the modified change detection result into multiple change detection tasks based on the chain thinking reasoning mechanism, generates a corresponding candidate solution for each change detection task, aggregates all the candidate solutions, and outputs the corresponding final change detection result.

8. The multi-modal large model anti-hallucination change detection device according to claim 7, characterized in that: include: A pixel-level information acquisition module is used to retrieve time feature differences and acquire pixel-level information between multiple remote sensing images based on the time feature differences; The iterative calibration module retrieves the preliminary change detection result and is used to iteratively calibrate the pixel-level information with the preliminary change detection result through the CVA-CICL mechanism to obtain the corresponding corrected change detection result.

9. A multimodal large model anti-hallucination change detection method, characterized in that: The method comprises a processor in which a program of the multimodal large model anti-hallucination change detection method as claimed in any one of claims 1 to 6 is run.

10. A storage medium, characterized in that: A program is stored for the multimodal large model anti-hallucination change detection method as described in any one of claims 1 to 6.

Images