Unsupervised multimodal large model training method and device, equipment and storage medium

Abstract

This invention discloses an unsupervised multimodal large-scale model training method, apparatus, device, and storage medium. The training method uses original remote sensing imagery and corresponding fuzzy text queries as a basis to generate prompts that drive the model to perform image understanding and spatial localization tasks. These prompts are then concatenated with the fuzzy text queries to form the first text. The original remote sensing imagery and the first text are used to construct training samples. After inputting these samples into the model, multiple sets of outputs containing inference text and predicted bounding boxes are generated. Based on the predicted bounding boxes, the original remote sensing imagery is cropped and reconstructed to generate verification prompts for semantic consistency determination. These prompts are then concatenated with the fuzzy text queries to form the second text. The reconstructed imagery and the second text are input into the model to complete semantic consistency self-verification. Based on the verification results, a soft reward function is constructed. The mean and standard deviation of the reward for each batch of samples are calculated to obtain the dominance function. The model parameters are updated iteratively until the soft reward mean converges, achieving self-evolution of the model's inference and fine-grained localization capabilities with zero manual annotation cost.

Description

Technical Field

[0001] This invention relates to unsupervised training methods, apparatus, devices, and storage media for large multimodal models, and belongs to the fields of computer data processing and artificial intelligence technology. Background Technology

[0002] In recent years, with the widespread adoption of the Transformer architecture, large language models (such as the GPT-5, Claude, and Gemini series) and multimodal models have made groundbreaking progress in natural language processing and computer vision. However, in complex tasks that require models to possess both deep logical reasoning and accurate visual localization capabilities, such as "please indicate areas where a large number of disaster victims can be quickly resettled and where emergency medical facilities can be deployed," existing training paradigms face a triple dilemma of data cost, model accuracy, and computational resources.

[0003] A pressing issue is how to achieve high-precision inference and localization training without human annotation, while avoiding the illusion problem of weak supervision and significantly reducing the GPU and computational resource consumption of reinforcement learning training. Shikra achieved end-to-end localization capabilities by converting bounding box coordinates into text tokens for autoregressive generation. However, such systems are highly dependent on the quality and format of the input data stream, requiring a large-scale, manually annotated database of image-text-coordinate triples (such as the RefCOCO dataset). In practical industrial applications (such as medical image analysis and industrial quality inspection), building such high-precision databases leads to enormous data storage costs and I / O throughput pressure. Furthermore, due to the subjective variability of manually annotated data, the system input is noisy, causing a sharp drop in inference accuracy when a specific ground truth signal is lacking, making it unable to effectively handle massive amounts of unannotated data in open scenarios.

[0004] To lower the data threshold, researchers have turned to weakly supervised learning, which uses only the final answer as a supervisory signal without providing intermediate reasoning processes or location coordinates. Some studies attempt to leverage the model's self-correcting capabilities, such as SelfEQ, which proposes using the model to generate multiple answers and calculate their consistency as a confidence metric. These systems suffer from severe logical errors and output instability (i.e., the "illusion" problem). Due to the lack of a calibration mechanism for intermediate visual features, the computational system is highly susceptible to "spurious correlation," meaning that although it outputs the correct classification label, the internal visual attention mechanism focuses on the wrong pixel region. At the system resource level, relying solely on "consistency" verification easily leads to the system converging to a "consistent but incorrect" local optimum, resulting in an ineffective waste of computational resources, and it cannot effectively correct the randomness error of coordinate prediction using a single text loss function (Loss). Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide an unsupervised multimodal large model training method, device, equipment and storage medium. By constructing a closed-loop computing architecture of "generation-pruning-self-verification", the parameters of the multimodal large language model are optimized and the consumption of computing resources is reduced under the condition of no manually labeled data. It is applicable to scenarios such as intelligent interpretation of remote sensing images and complex logical question answering.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] This invention proposes a self-verifying unsupervised training method for large multimodal models, the method comprising:

[0008] Based on the original remote sensing image and the corresponding fuzzy text query, a prompt instruction is generated to drive the multimodal large language model to perform image understanding and spatial localization tasks. The prompt instruction and the fuzzy text query are concatenated as the first text.

[0009] For the current training batch, the original remote sensing image and the corresponding first text are input as a sample into the multimodal large language model. The group relative strategy optimization algorithm is used to sample and generate several different candidate output sequences. Each candidate output sequence includes the inference text and the predicted bounding box corresponding to the sample.

[0010] The original remote sensing image is cropped and visually reconstructed based on the predicted bounding boxes in each candidate output sequence to obtain the reconstructed remote sensing image.

[0011] Based on the reconstructed remote sensing image and the fuzzy text query corresponding to the original remote sensing image, a verification prompt instruction is generated to drive the multimodal large language model to make a semantic consistency judgment on the fuzzy text query on the reconstructed remote sensing image. The verification prompt instruction and the fuzzy text query are concatenated to form the second text.

[0012] The reconstructed remote sensing images and the corresponding second text are input into a multimodal large language model for self-verification of semantic consistency.

[0013] Based on the self-validation results, a soft reward function is constructed, and the mean and standard deviation of the soft reward for all samples in the current training batch are calculated. Then, the advantage function of each sample is calculated, and the parameters of the multimodal large language model are updated using the group relative strategy optimization algorithm before entering the next training batch. When the mean of the soft reward tends to converge, the training ends.

[0014] As a preferred training method, the original remote sensing image is cropped and visually reconstructed based on the predicted bounding boxes in each output group to obtain the reconstructed remote sensing image, as detailed below:

[0015] For candidate output sequences , and the predicted bounding boxes Converting vertex coordinates to the original remote sensing image Pixel coordinates in;

[0016] Based on pixel coordinates from the original remote sensing image Crop a partial image segment from the middle ;

[0017] For local image fragments Scaling or padding is performed to adapt to the input size of the multimodal large language model, resulting in the reconstructed remote sensing image.

[0018] As a preferred training method, the soft reward function includes a format reward and a semantic consistency reward, as shown in the following formula:

[0019] ,

[0020] in, Candidate output sequence The corresponding soft reward value; As a format reward, if the multimodal large language model outputs coordinates in a compliant format, a fixed base positive reward is given; otherwise, a penalty is imposed. These are the weighting coefficients; For semantic consistency reward, the probability value of the multimodal large language model output during semantic consistency self-verification is selected as the semantic consistency reward value.

[0021] As a preferred training method, the parameter update of the multimodal large language model using the group relative policy optimization algorithm is as follows:

[0022] We construct a policy optimization objective function using the advantage function, and update the parameters of the multimodal large language model by maximizing the policy optimization objective function. The policy optimization objective function is expressed as:

[0023] ,

[0024] in, This represents the objective function for policy optimization. Represents the original remote sensing image. This represents a fuzzy text query corresponding to the original remote sensing image. For the distribution of training data; This is the multimodal large language model before parameter updates. For the multimodal large language model after parameter updates, For reference to multimodal large language models; This represents the number of candidate output sequences obtained by downsampling from the same input sample. For the first 1 candidate output sequence Candidate output sequence The number of tokens, For the token location index, This is a numerical truncation function; This is the cutting factor. For KL regularization weights, In a given The KL divergence term calculated based on the output distribution under the given conditions; The advantage function is used to characterize the candidate output sequence. The relative merits and demerits within the group formed by all candidate output sequences;

[0025] The strategy ratio term is defined as follows:

[0026] ,

[0027] The advantage function Calculated based on normalized group rewards, defined as:

[0028] ,

[0029] in, Candidate output sequence In the Output token at each position; Candidate output sequence In the The historical token subsequence preceding the i-th position is used to represent the i-th position under the autoregressive generation condition. The conditional probability dependency of each token; These are the mean and standard deviation of the soft rewards for the same input sample, respectively.

[0030] This invention also proposes a self-verifying unsupervised multimodal large model training device, the device comprising:

[0031] The data preprocessing module is used to generate prompts based on the original remote sensing images and the corresponding fuzzy text queries, which are used to drive the multimodal large language model to perform image understanding and spatial localization tasks. The prompts and fuzzy text queries are concatenated to form the first text.

[0032] The inference generation module is used to input the original remote sensing data, images and corresponding first text as a sample into the multimodal large language model for the current training batch, and use the group relative strategy optimization algorithm to sample and generate several different candidate output sequences. Each candidate output sequence includes the inference text and predicted bounding box corresponding to the sample.

[0033] The visual reconstruction module is used to crop and visually reconstruct the original remote sensing image based on the predicted bounding boxes in each candidate output sequence to obtain the reconstructed remote sensing image.

[0034] The closed-loop verification module is used to generate verification prompts based on the reconstructed remote sensing image and the fuzzy text query corresponding to the original remote sensing image. These prompts are used to drive the multimodal large language model to make a semantic consistency judgment on the fuzzy text query in the reconstructed remote sensing image. The verification prompts are then concatenated with the fuzzy text query to form the second text.

[0035] The reconstructed remote sensing images and the corresponding second text are input into a multimodal large language model for self-verification of semantic consistency.

[0036] The parameter optimization module is used to construct a soft reward function based on the self-validation results, calculate the mean and standard deviation of the soft reward for all samples in the current training batch, and then calculate the advantage function for each sample. The parameters of the multimodal large language model are updated using the group relative strategy optimization algorithm and the model enters the next training batch. Training ends when the mean of the soft reward tends to converge.

[0037] This invention also proposes an intelligent interpretation method for remote sensing images based on a multimodal large model, the method comprising:

[0038] The remote sensing image and the corresponding fuzzy text query are input into a trained multimodal large language model to obtain the interpretation result corresponding to the fuzzy text query.

[0039] The trained multimodal large language model is obtained based on the self-verification-based unsupervised multimodal large model training method.

[0040] The present invention also proposes a computer device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the computer program to implement the steps of the self-verification-based unsupervised multimodal large model training method described above.

[0041] The present invention also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the self-verifying unsupervised multimodal large model training method described above.

[0042] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0043] 1. This invention eliminates the reliance on expensive bounding box (Bbox) annotation data (low cost and high scalability). Existing data processing typically relies on large-scale manually annotated "image-text-coordinate" triple databases (Ground Truth) to maintain operational accuracy, resulting in huge data acquisition costs and storage I / O pressure. By introducing a "self-verifying" closed-loop computation mechanism, it is possible to directly utilize raw, unstructured image and text data streams for parameter iteration. This enables the computer system to perform self-supervised evolution using massive amounts of unannotated data from the Internet, significantly reducing the system deployment's dependence on high-precision annotation data, while also reducing the computational load of data preprocessing caused by handling complex annotation formats.

[0044] 2. This invention resolves the contradiction between generation capability and understanding capability. In multimodal computing, the accuracy of the discrimination module is generally higher than that of the generation module. This invention constructs a feedback loop based on probability distribution, directly utilizing its own high-precision discrimination capability (i.e., the semantic understanding probability of the cropped image content) as a supervision signal to calibrate the relatively low-precision coordinate generation module. This internal signal calibration mechanism does not require the introduction of an additional external reward model, achieving dynamic approximation of semantic understanding capability by the localization generation capability, and improving the accuracy of the output signal.

[0045] 3. This invention enhances the reasoning and localization capabilities (semantic consistency) for fuzzy semantic queries. Traditional hard loss functions based on IoU (Intersection over Union) are prone to convergence difficulties or logical errors when processing unstructured fuzzy instructions such as "suitable placement area" due to the non-uniqueness of the standard answer. This invention employs a soft reward signal processing mechanism based on semantic consistency. As long as the image features of the cropped region can trigger a high-confidence response semantically, positive feedback is given. This processing method enhances the computer system's ability to resolve fuzzy instructions in open scenes, enabling the system to output logically consistent and accurately located calculation results even when faced with non-standardized inputs, avoiding the "collapse" or "illusion" phenomena of system output.

[0046] 4. Compared to traditional PPO (Proximal Policy Optimization)-based system architectures that require loading a separate value evaluation network (Value Model), this invention employs a Group Relative Policy Optimization (GRPO) computational architecture. This architecture estimates the baseline by calculating the relative advantage of samples within a group, eliminating the GPU memory requirement for a separate value network model. In practical deployments, this directly reduces the GPU memory (VRAM) usage during model training and avoids the overhead of value model synchronization communication during multi-GPU distributed computing, enabling the system to achieve more efficient and stable parallel computing on large-scale parameter models. Attached Figure Description

[0047] Figure 1 This is a flowchart of a self-verifying unsupervised multimodal large model training method proposed in this invention;

[0048] Figure 2 This is a schematic diagram of a self-verifying unsupervised multimodal large model training method proposed in this invention.

[0049] Figure 3 This is a schematic diagram illustrating an application scenario of an embodiment of the present invention. Detailed Implementation

[0050] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0051] like Figure 1 and Figure 2As shown, this invention proposes a self-verification-based unsupervised multimodal large model training method. By constructing an automated data processing closed loop of "generation-clipping-self-verification," the semantic understanding capability of the multimodal large model is used as a feedback signal source to guide the optimization of generation parameters. Utilizing the Group Relative Policy Optimization (GRPO) computational architecture, the model parameters can achieve self-iteration and convergence using only raw images and unstructured text queries, without requiring externally labeled bounding box (GT) data. The specific steps are as follows:

[0052] Step S1: Input data preprocessing and instruction construction

[0053] Based on the original remote sensing images and corresponding fuzzy text query (For example: find suitable areas in the image for resettling disaster victims), and generate prompts to drive the multimodal large language model to perform image understanding and spatial localization tasks.

[0054] In this embodiment, the prompt instructions received by the model are shown in the following example:

[0055] You are a remote sensing image understanding and reasoning assistant. Your task is to locate target objects in an image based on a given text description and output their corresponding bounding box coordinates. Please follow these requirements: 1) Text understanding: Identify target objects based on the text description; 2) Output method: If a target exists, output the bounding box coordinates in the following format: If no target exists, output an empty list "[]"; if multiple targets exist, output multiple bounding boxes in the following format. 3) Coordinate Specifications: Coordinate values ​​must be pixel integers, and the output bounding box should be the smallest rectangular area tightly enclosing the target; 4) Output Order: Please first provide the reason for the positioning (in the following order). <think>(label representation), and then output the final location result (in terms of labels). <answer>(Label representation). Please complete the above location task based on the image content.

[0056] Driven by the above prompts, the model outputs two key types of content: (1) inference chain : Explain the logical basis for finding the target; (2) Location prediction: Output the bounding box in coordinate form. .

[0057] Step S2: Inference generation based on group sampling

[0058] Utilizing current multimodal large language models For the same input Sample generation Different output groups Each output Includes corresponding reasoning text and predicted bounding boxes This step utilizes the characteristics of the GRPO algorithm to find the optimal solution by exploring different inference paths and localization results.

[0059] Step S3: Visual Reconstruction and Cropping

[0060] For each predicted bounding box generated in step S2 Perform image processing operations:

[0061] (1) Coordinate resolution: Convert the normalized coordinates into pixel coordinates of the original image;

[0062] (2) Region cropping: Based on pixel coordinates, cropping is performed on the original remote sensing image. Crop a local image segment ;

[0063] (3) Pretreatment: The image is scaled or padded to fit the model's visual encoder input size, resulting in the reconstructed visual input. Alternatively, visual prompting can be used. In this mode, instead of physically segmenting the image tensor, highlight masks, bounding box lines, or indicator arrows are overlaid on the original image by modifying pixel data. The processed enhanced image data stream is then input into step S4, where the model calculates the probability of whether the "marked region matches the query".

[0064] Step S4: Self-verification based on semantic consistency

[0065] This step implements the core data processing for unsupervised optimization, transforming the clipping image obtained in step S3. and raw text query Reassemble and reconstruct the verification prompt instructions. The input is fed into the model itself (at this time, the model acts as a validator, in inference mode or using its frozen copy).

[0066] In this embodiment, the verification prompt instruction received by the model is shown in the following example:

[0067] Please determine whether the image region shown matches the description in the user's query above, and perform objective verification. Please analyze and provide your reasoning (using...). <think>(labels are used to indicate this), followed by the final judgment result (in the form of labels). <answer>(Label indicates), the result is limited to Yes or No.

[0068] The above verification prompts are used to require the model to perform tests in local image regions. above query for raw text The model determines semantic consistency and outputs a binary result. After processing the input, it does not directly take the generated text result, but extracts the Logits value corresponding to the 'Yes' token in the output vocabulary, denoted as . .

[0069] Step S5: Calculation of soft rewards

[0070] Construct a soft reward function, which consists of two parts:

[0071] (1) Format rewards If the model outputs coordinates in a format that conforms to the specifications, a fixed positive reward is given; otherwise, a penalty is imposed.

[0072] (2) Semantic consistency reward : Directly use the probability value obtained in step S4 As a soft reward signal, i.e. ;

[0073] The semantic consistency reward can be configured as a discrete signal processing mode (Hard Reward), meaning that a high-level (full score) reward signal is generated only when the confidence threshold of the model output exceeds a preset limit (e.g., 0.9); otherwise, a low-level (zero score) signal is generated. This discretization reduces the system's sensitivity to small probability fluctuations.

[0074] Soft reward for each sample ,in, These are the weighting coefficients.

[0075] Step S6: Group-based relative strategy optimization update

[0076] Calculate this group Average reward of each sample and standard deviation Calculate the advantage function for each sample: .

[0077] Based on this, the aforementioned Group Relative Policy Optimization (GRPO) algorithm is used for parameter updates. This involves constructing a policy optimization objective function using the advantage function, ensuring that samples with higher advantages receive higher weights during the update process. The GRPO policy objective function can be expressed as:

[0078] ,

[0079] in, Represents the original remote sensing image. This represents a fuzzy text query corresponding to the original remote sensing image. For the distribution of training data; For the old strategy model, For reference strategy model; This represents the number of candidate outputs obtained by downsampling from the same input. For the first 1 candidate output sequence Candidate output sequence The number of tokens. Indexed by token; This is the cutting factor. For KL regularization weights, In a given KL divergence term calculated based on output distribution under given conditions; The advantage value is used to characterize the relative merit of a candidate output within a group. In this embodiment, It is obtained by sample-level reward normalization and aligned to each token in the sequence, therefore and Irrelevant.

[0080] Strategy Ratio Item Defined as:

[0081] ,

[0082] Advantage function Calculated based on normalized group rewards, defined as:

[0083] ,

[0084] The multimodal large language model is updated by maximizing the objective function of the above GRPO strategy. parameters This makes multimodal large language models more likely to generate candidate outputs with higher advantages, thereby improving the consistency between the localization results and the query semantics.

[0085] Alternatively, the module group's relative policy optimization algorithm can also be configured to execute the Proximal Policy Optimization (PPO) protocol, requiring additional memory allocation to instantiate the Critic network; or it can be configured to execute variant protocols such as Group Sequence Policy Optimization (GSPO) and Group Variance Policy Optimization (GVPO). Furthermore, a Direct Preference Optimization (DPO) architecture can be employed, directly fine-tuning the end-to-end parameters of the policy network by constructing a "high-confidence-low-confidence" sample pair data stream, skipping the explicit reinforcement learning reward calculation stage.

[0086] The default output is the coordinate data of a rectangular bounding box. For different application scenarios, it can be reconstructed to output the center point coordinates (the verification module automatically extracts a fixed radius region), polygon vertex sets, or pixel-level segmentation masks. Regardless of the geometric data encoding format used, the corresponding spatial positioning parameters are iteratively optimized through the aforementioned closed-loop verification mechanism.

[0087] like Figure 3 As shown, the "I will draw bounding boxes around the entire baseball field" generated by the model in the Think stage is not only the output reason, but also the self-verifying pre-logic.

[0088] Closed-loop operation: The visual reconstruction module performs signal cropping on the original image based on the coordinates shown in the Bbox in the figure, generating a partial view that only contains the area within the green box.

[0089] Self-verification process: Input the partial view back into the model to verify its semantic consistency with "community team activity venue".

[0090] Optimization logic: If the model gives a high-probability "Yes" signal during the validation phase, the Logits value is converted into a soft reward, and the parameters are updated using the GRPO algorithm, thereby achieving autonomous evolution without manual annotation.

[0091] This invention has significant application value in the fields of intelligent interpretation of remote sensing images and emergency management. For example... Figure 3 As shown, this invention can handle complex geospatial queries, not only identifying large sports venues such as baseball fields, but also accurately locating critical facilities such as hospital helipads based on specific business logic such as emergency medical evacuation. This capability makes it very suitable for application in natural language-driven satellite remote sensing retrieval. In emergency tasks such as earthquake relief, it can automatically locate compliant areas in massive amounts of imagery based on fuzzy descriptive instructions (such as "find areas that can accommodate disaster victims and support the deployment of emergency medical facilities"), eliminating the reliance on traditional and expensive bounding box annotation data.

[0092] In complex logic-driven urban planning and intelligent security scenarios, this invention enables deep semantic referential understanding. It not only focuses on the visual features of objects but also ensures that the positioning results conform to complex logical requirements through autonomous thinking, such as identifying "venues suitable for team-based sports activities" in an example image. This semantic consistency-based processing approach allows the system to output logically consistent and accurately positioned results when processing urban planning indicator monitoring, non-standard functional area delineation, and specific behavioral logic recognition in long-term surveillance videos, effectively avoiding the "illusion" phenomenon common in traditional multimodal models.

[0093] Furthermore, it can also be applied in non-remote sensing scenarios, such as self-supervised assisted diagnosis of medical images, or automated discovery and closed-loop verification of non-standard defects lacking standard ground truth in industrial quality inspection, achieving self-evolution of positioning accuracy and understanding ability through internal feedback loops.

[0094] This invention also proposes a self-verifying unsupervised multimodal large model training device, comprising:

[0095] The data preprocessing module is used to generate prompts based on the original remote sensing images and the corresponding fuzzy text queries, which are used to drive the multimodal large language model to perform image understanding and spatial localization tasks. The prompts and fuzzy text queries are concatenated to form the first text.

[0096] The inference generation module is used to input the original remote sensing image and the corresponding first text as a sample into the multimodal large language model for the current training batch, and use the group relative strategy optimization algorithm to sample and generate several different candidate output sequences. Each candidate output sequence includes the inference text and the predicted bounding box corresponding to the sample.

[0097] The visual reconstruction module is used to crop and visually reconstruct the original remote sensing image based on the predicted bounding boxes in each candidate output sequence to obtain the reconstructed remote sensing image.

[0098] The closed-loop verification module is used to generate verification prompts based on the reconstructed remote sensing image and the fuzzy text query corresponding to the original remote sensing image. These prompts are used to drive the multimodal large language model to make a semantic consistency judgment on the fuzzy text query in the reconstructed remote sensing image. The verification prompts are then concatenated with the fuzzy text query to form the second text.

[0099] The reconstructed remote sensing images and the corresponding second text are input into a multimodal large language model for self-verification of semantic consistency.

[0100] The parameter optimization module is used to construct a soft reward function based on the self-validation results, calculate the mean and standard deviation of the soft reward for all samples in the current training batch, and then calculate the advantage function for each sample. The parameters of the multimodal large language model are updated using the group relative strategy optimization algorithm and the model enters the next training batch. Training ends when the mean of the soft reward tends to converge.

[0101] This invention also proposes an intelligent interpretation method for remote sensing images based on a multimodal large model. The remote sensing image and the corresponding fuzzy text query are input into a trained multimodal large language model to obtain the interpretation result corresponding to the fuzzy text query. The trained multimodal large language model is obtained based on the aforementioned self-verification-based unsupervised multimodal large model training method.

[0102] Based on the same inventive concept, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the aforementioned self-verifying unsupervised multimodal large model training method.

[0103] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the aforementioned self-verifying unsupervised multimodal large model training method.

[0104] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0105] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0106] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0107] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0108] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.< / answer> < / think> < / answer> < / think>

Claims

1. A self-verifying unsupervised training method for large multimodal models, characterized in that, The method includes: Based on the original remote sensing image and the corresponding fuzzy text query, a prompt instruction is generated to drive the multimodal large language model to perform image understanding and spatial localization tasks. The prompt instruction and the fuzzy text query are concatenated as the first text. For the current training batch, the original remote sensing image and the corresponding first text are input as a sample into the multimodal large language model. The group relative strategy optimization algorithm is used to sample and generate several different candidate output sequences. Each candidate output sequence includes the inference text and the predicted bounding box corresponding to the sample. The original remote sensing image is cropped and visually reconstructed based on the predicted bounding boxes in each candidate output sequence to obtain the reconstructed remote sensing image. Based on the reconstructed remote sensing image and the fuzzy text query corresponding to the original remote sensing image, a verification prompt instruction is generated to drive the multimodal large language model to make a semantic consistency judgment on the fuzzy text query on the reconstructed remote sensing image. The verification prompt instruction and the fuzzy text query are concatenated to form the second text. The reconstructed remote sensing images and the corresponding second text are input into a multimodal large language model for self-verification of semantic consistency. Based on the self-validation results, a soft reward function is constructed, and the mean and standard deviation of the soft reward for all samples in the current training batch are calculated. Then, the dominance function of each sample is calculated, and the parameters of the multimodal large language model are updated using the group relative strategy optimization algorithm before entering the next training batch. When the mean of the soft reward tends to converge, the training ends. The soft reward function includes a format reward and a semantic consistency reward, as shown in the following formula: ， in, Candidate output sequence The corresponding soft reward value; As a format reward, if the multimodal large language model outputs coordinates in a compliant format, a fixed base positive reward is given; otherwise, a penalty is imposed. These are the weighting coefficients; For semantic consistency reward, the probability value of the multimodal large language model output during semantic consistency self-verification is selected as the semantic consistency reward value.

2. The self-verifying unsupervised multimodal large model training method according to claim 1, characterized in that, The original remote sensing image is cropped and visually reconstructed based on the predicted bounding boxes in each candidate output sequence to obtain the reconstructed remote sensing image, as detailed below: For candidate output sequences , and the predicted bounding boxes Converting vertex coordinates to the original remote sensing image Pixel coordinates in; Based on pixel coordinates from the original remote sensing image Crop a partial image segment from the middle ; For local image fragments Scaling or padding is performed to adapt to the input size of the multimodal large language model, resulting in the reconstructed remote sensing image.

3. The self-verifying unsupervised multimodal large model training method according to claim 2, characterized in that, The parameters of the multimodal large language model are updated using the group relative strategy optimization algorithm as follows: We construct a policy optimization objective function using the advantage function, and update the parameters of the multimodal large language model by maximizing the policy optimization objective function. The policy optimization objective function is expressed as: ， in, This represents the objective function for policy optimization. Represents the original remote sensing image. This represents a fuzzy text query corresponding to the original remote sensing image. For the distribution of training data; This is the multimodal large language model before parameter updates. For the multimodal large language model after parameter updates, For reference to multimodal large language models; This represents the number of candidate output sequences obtained by downsampling from the same input sample. For the first 1 candidate output sequence Candidate output sequence The number of tokens, For the token location index, This is a numerical truncation function; This is the cutting factor. For KL regularization weights, In a given The KL divergence term calculated based on the output distribution under the given conditions; The advantage function is used to characterize the candidate output sequence. The relative merits and demerits within the group formed by all candidate output sequences; The strategy ratio term is defined as follows: ， The advantage function Calculated based on normalized group rewards, defined as: ， in, Candidate output sequence In the Output token at each position; Candidate output sequence In the The historical token subsequence preceding the i-th position is used to represent the i-th position under the autoregressive generation condition. The conditional probability dependency of each token; These represent the mean and standard deviation of the soft rewards for the same input sample, respectively.

4. A self-verifying unsupervised multimodal large model training device, characterized in that, The device includes: The data preprocessing module is used to generate prompts based on the original remote sensing images and the corresponding fuzzy text queries, which are used to drive the multimodal large language model to perform image understanding and spatial localization tasks. The prompts and fuzzy text queries are concatenated to form the first text. The inference generation module is used to input the original remote sensing image and the corresponding first text as a sample into the multimodal large language model for the current training batch, and use the group relative strategy optimization algorithm to sample and generate several different candidate output sequences. Each candidate output sequence includes the inference text and the predicted bounding box corresponding to the sample. The visual reconstruction module is used to crop and visually reconstruct the original remote sensing image based on the predicted bounding boxes in each candidate output sequence to obtain the reconstructed remote sensing image. The closed-loop verification module is used to generate verification prompts based on the reconstructed remote sensing image and the fuzzy text query corresponding to the original remote sensing image. These prompts are used to drive the multimodal large language model to make a semantic consistency judgment on the fuzzy text query in the reconstructed remote sensing image. The verification prompts are then concatenated with the fuzzy text query to form the second text. The reconstructed remote sensing images and the corresponding second text are input into a multimodal large language model for self-verification of semantic consistency. The parameter optimization module is used to construct a soft reward function based on the self-validation results, calculate the soft reward mean and standard deviation of all samples in the current training batch, and then calculate the dominance function of each sample. The parameters of the multimodal large language model are updated using the group relative strategy optimization algorithm and the model enters the next training batch. Training ends when the soft reward mean tends to converge. The soft reward function includes a format reward and a semantic consistency reward, as shown in the following formula: ， in, Candidate output sequence The corresponding soft reward value; As a format reward, if the multimodal large language model outputs coordinates in a compliant format, a fixed base positive reward is given; otherwise, a penalty is imposed. These are the weighting coefficients; For semantic consistency reward, the probability value of the multimodal large language model output during semantic consistency self-verification is selected as the semantic consistency reward value.

5. A method for intelligent interpretation of remote sensing images based on a multimodal large model, characterized in that, The method includes: The remote sensing image and the corresponding fuzzy text query are input into a trained multimodal large language model to obtain the interpretation result corresponding to the fuzzy text query. The trained multimodal large language model is obtained based on the self-verification-based unsupervised multimodal large model training method described in any one of claims 1 to 3.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the self-verifying unsupervised multimodal large model training method as described in any one of claims 1 to 3.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the self-verifying unsupervised multimodal large model training method as described in any one of claims 1 to 3.

Images