Sample construction method, multimodal large language model fine-tuning method and related device

Abstract

The present application relates to the technical field of artificial intelligence, and discloses a sample construction method, a multi-modal large language model fine-tuning method and related devices. The sample construction method comprises: obtaining multi-modal data and an original prompt word; based on the original prompt word, performing semantic understanding on the multi-modal data to obtain a preliminary description text corresponding to the multi-modal data; performing structured data extraction on the multi-modal data to obtain structured objective fact data; performing fact difference detection on the preliminary description text and the objective fact data; and if a fact difference is detected, constructing a preference sample for multi-modal large language model fine-tuning based on the original prompt word, the preliminary description text and the objective fact data. The present application can automatically generate high-quality preference samples for model preference alignment training, greatly reducing the model alignment cost and avoiding the subjectivity and inconsistency that may be introduced by manual annotation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a sample construction method, a multimodal large language model fine-tuning method, and related devices. Background Technology

[0002] Multimodal large language models have demonstrated outstanding capabilities in text and image understanding and generation, but they commonly suffer from the illusion of fact problem. This means that when generating text, the models tend to deviate from the objective facts in multimodal data (such as images and videos) and fabricate non-existent information (e.g., misidentifying equipment or scores in game scenes). To eliminate this illusion, the industry typically requires fine-tuning the models.

[0003] Existing model fine-tuning heavily relies on high-quality sample datasets. Current techniques primarily depend on manual ranking of multiple model outputs when constructing training datasets for fine-tuning. For domains with massive amounts of data and rapid content iteration, such as game streaming and short videos, manual annotation is not only costly and time-consuming but also subject to subjectivity and inconsistency, failing to meet the needs of large-scale fine-tuning. While some methods exist for selecting fine-tuning data through rules or simple models, they struggle to accurately identify hidden factual errors in model outputs, resulting in inconsistent quality of the constructed training data and failing to effectively guide the model in correcting factual illusions. Summary of the Invention

[0004] The main objective of this invention is to provide a sample construction method, a multimodal large language model fine-tuning method, and related devices, aiming to solve the technical problems of high cost and inconsistent quality of sample data construction for existing model fine-tuning training.

[0005] To achieve the above objectives, the present invention provides a sample construction method, the sample construction method comprising: Acquire multimodal data and raw prompt words; Based on the original prompt words, semantic understanding is performed on the multimodal data to obtain the preliminary descriptive text corresponding to the multimodal data; Structured data extraction is performed on the multimodal data to obtain structured objective fact data; Perform fact discrepancy detection between the preliminary description text and the objective fact data; If a factual discrepancy is detected, then a preference sample for fine-tuning the multimodal large language model is constructed based on the original prompt words, the preliminary description text, and the objective factual data.

[0006] Optionally, in a first implementation of the first aspect of the present invention, the objective factual data includes textual information, key object information, and event progression stage information in the multimodal data.

[0007] Optionally, in a second implementation of the first aspect of the present invention, the step of performing fact difference detection between the preliminary description text and the objective fact data includes: Obtain preset detection prompts for fact discrepancy detection; The detection prompts, the preliminary description text, and the objective fact data are input into a pre-set large language model to perform fact discrepancy detection, and the fact discrepancy detection results are output.

[0008] Optionally, in the third implementation of the first aspect of the present invention, if a factual discrepancy is detected, constructing a preference sample for fine-tuning the multimodal large language model based on the original prompt word, the preliminary description text, and the objective factual data includes: If a factual discrepancy is detected, the preliminary description text is used as a negative sample representing the error generation tendency of the multimodal large language model, and a positive sample representing the correct generation tendency of the multimodal large language model is generated based on the objective factual data. The original prompt words, the positive samples, and the corresponding negative samples are combined to obtain triplet preference samples for fine-tuning the multimodal large language model.

[0009] Optionally, in a fourth implementation of the first aspect of the present invention, generating positive samples representing the correct generation tendency of the multimodal large language model based on the objective factual data includes: Mapping the objective factual data to a preset text description template yields positive samples representing the correct generation tendency of the multimodal large language model; or By using customized prompt words to instruct the generative large language model to generate factual description text corresponding to the objective factual data, the factual description text is used as a positive sample representing the correct generation tendency of the multimodal large language model.

[0010] Optionally, in a fifth implementation of the first aspect of the present invention, the sample construction method further includes: If no factual discrepancies are detected, the preliminary description text is determined to be high-quality text, and the preliminary description text and the original prompt words are used as samples for model-supervised fine-tuning.

[0011] A second aspect of the present invention provides a method for fine-tuning a multimodal large language model, the method comprising: Obtain multimodal data samples and preference samples, wherein the preference samples include original prompt words, positive samples and negative samples, and the preference samples are obtained using the sample construction method described in any of the above. The original prompt words, the positive samples, the negative samples, and the multimodal data samples are input into the multimodal large language model to be fine-tuned, and the first probability of generating the positive samples and the second probability of generating the negative samples are output. Based on the difference between the first probability and the second probability, a loss function is constructed, and based on the loss function, the parameters of the multimodal large language model are updated through backpropagation.

[0012] A third aspect of the present invention provides a sample construction apparatus, the sample construction apparatus comprising: The acquisition module is used to acquire multimodal data and raw prompt words; The generation module is used to perform semantic understanding on the multimodal data based on the original prompt words to obtain preliminary description text corresponding to the multimodal data; The extraction module is used to extract structured data from the multimodal data to obtain structured objective fact data; The detection module is used to perform fact difference detection between the preliminary description text and the objective fact data; The construction module is used to construct preference samples for fine-tuning the multimodal large language model based on the original prompt words, the preliminary description text, and the objective fact data if a factual discrepancy is detected.

[0013] A fourth aspect of the present invention provides a computer device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor invokes the instructions in the memory to cause the computer device to perform the above-described sample construction method and / or multimodal large language model fine-tuning method.

[0014] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the above-described sample construction method and / or the multimodal large language model fine-tuning method.

[0015] This invention provides a sample construction method, a multimodal large language model fine-tuning method, and related apparatus. In sample construction, multimodal data and original prompts are first acquired. Then, based on the instructions of the original prompts, semantic understanding is performed on the multimodal data to obtain preliminary descriptive text corresponding to the multimodal data. Simultaneously, structured data extraction is performed on the multimodal data input to obtain various objective factual data within the multimodal data. Then, factual difference detection is performed between the preliminary descriptive text, which may contain factual illusions, and the objective factual data. If a factual difference is detected, it indicates that the preliminary descriptive text contains factual illusions and can be used as negative sample data for model preference training, while the objective factual data can be used as positive sample data for model preference training. This embodiment cleverly utilizes the conflict between the model's own erroneous output (illusions) and external multi-source objective facts to automatically generate high-quality negative and positive samples, completely eliminating the expensive manual preference labeling process, establishing a fully automated pipeline for fine-tuning sample production, and greatly reducing the cost of model alignment. The construction benchmark for preference samples is derived from structured facts extracted by expert models such as text recognition and object detection, rather than human subjective judgment. This ensures that the positive samples in the constructed preference samples have absolute factual correctness and can be used to accurately eliminate factual illusions in the model. In this embodiment, preference optimization training is performed directly through the constructed preference samples, thereby adjusting the model's own parameter weights, enabling it to learn to focus on objective facts, and thus actively reduce illusions in future reasoning, achieving self-evolution of the model's capabilities. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of one embodiment of the sample construction method in this invention; Figure 2 This is a schematic diagram of an embodiment of the multimodal large language model fine-tuning method in this invention; Figure 3 This is a schematic diagram of one embodiment of the sample construction device in this invention; Figure 4 This is a schematic diagram of one embodiment of the computer device in this invention. Detailed Implementation

[0017] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0018] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 One embodiment of the sample construction method in this invention includes: 101. Obtain multimodal data and raw prompt words; When applying multimodal large language models to real-time streaming scenarios such as e-sports commentary and virtual interactive entertainment, the primary challenge is the multi-source heterogeneity of the data and the extremely high timeliness requirements. Multimodal data not only contains static image pixel information but may also include visual elements that dynamically change over time. Traditional model training often uses manually selected and cleaned static image datasets, which deviates significantly from the complex, dynamic, and highly redundant real-world data distribution encountered in actual inference. If a random image without context is simply picked without accompanying instructions relevant to a specific business scenario, the text generated by the model will be aimless and unable to accurately trigger factual illusions in specific vertical domains (such as equipment recognition, status determination, and skill analysis in games). Without accurate exposure of these illusions, it is impossible to construct effective correction samples.

[0019] In this step, acquiring multimodal data is not a simple image reading process, but a complex streaming media processing mechanism. In e-sports live streaming scenarios, the system pulls real-time video streams via RTMP or WHIP protocols. To balance computational cost and data diversity, a keyframe extraction strategy based on scene change detection is adopted: the histogram difference or perceptual hash value of adjacent video frames is calculated, and when the difference exceeds a preset threshold, it is determined to be a scene switch, and the current frame is extracted as the multimodal data source. After extraction, a strict preprocessing pipeline must be performed: first, aspect ratio adaptation is performed. For the 16:9 or 21:9 widescreen ratio commonly found in game UIs, a filling strategy that maintains the aspect ratio is adopted (such as filling the sides with black borders), uniformly scaling it to the model's standard input resolution (such as 448x448 pixels) to avoid image distortion that could cause subsequent object detection models to fail; then, pixel value normalization is performed, linearly mapping the pixel values ​​of the RGB channels from 0-255 to the [-1,1] range to match the data distribution during the visual encoder's pre-training.

[0020] The original prompts are pre-designed sets of highly targeted text instructions, which are dynamically generated by a template engine and contain a rich variety of variations. For example, they include not only "describe the scene" but also diverse instructions such as "analyze the current situation in the tone of a professional commentator" and "focus on the state of the character in the center of the scene." The purpose is to "pressure" the multimodal large language model to be fine-tuned from different semantic perspectives, maximizing the exposure of its cognitive defects.

[0021] For example, suppose the acquired multimodal data is a high-resolution screenshot of a massively multiplayer online role-playing game. The screenshot contains complex dark-style background textures, a mage character played by the player in the center of the screen, dense skill effects and lighting effects around the player, and various status UI elements scattered in the four corners of the screen. The original prompts are randomly selected from a command pool, and could be set as: "As a seasoned game commentator, please provide a detailed report on the player's equipment appearance, health and mana status, and current dungeon stage in the current scene." 102. Based on the original prompt words, perform semantic understanding on the multimodal data to obtain the preliminary description text corresponding to the multimodal data; Multimodal large language models awaiting fine-tuning are typically composed of a visual encoder and a large language model connected via a projection matrix. The root cause of the "factual illusion" generated by multimodal large language models lies in two levels of misalignment: the first level is visual perception misalignment, where the visual encoder, when extracting features, often loses crucial fine-grained features (such as small-font UI numbers or specific textures on item edges) due to numerous occlusions, sudden changes in lighting, or special effects interference in the image; the second level is language generation misalignment, where the language model, when generating text through autoregressive decoding, does not truly "understand" the image, but rather relies on the high-frequency co-occurrence probabilities in its pre-training corpus for "chaining." For example, in game corpora, "mage," "release," and "fireball" frequently co-occur; when visual features are blurred, the language model will directly "fill in" content that does not actually exist in the current image based on statistical probability. The purpose of this step is precisely to actively and fully expose this inherent defect determined by the model's inherent parameter structure through "zero-constraint" reasoning.

[0022] The preprocessed multimodal data tensor from step 101 is input into the visual encoder, and image feature sequences are extracted through a multi-layer self-attention mechanism. Simultaneously, the original prompt words are converted into text feature sequences through a word embedding layer. Both are then aligned using a cross-attention mechanism for deep multimodal feature integration. Crucially, no external knowledge graph retrieval enhancements are introduced, nor are any rule-based factual constraints applied; the model must reason with an absolutely "zero-sample" state. In the decoding stage, a kernel sampling strategy is employed to maintain a certain probability distribution stability while ensuring the diversity of generated text, allowing the model to freely output a natural language descriptive text (denoted as T). raw This is the initial description text. This initial description text must truthfully and authentically reflect the model's cognitive level under the current parameter conditions.

[0023] For example, suppose the player is actually holding a "Wrath of the Abyss Arcane Staff" in the actual game, with a red gem embedded at the top, and the UI displays 2500 health. However, because the staff is obscured by a large area of ​​fire effects and the UI font is small, the initial description text output after semantic understanding by a multimodal large language model is: "The player is holding an Ice Staff and preparing to cast a spell. Currently, their remaining health is very low, approximately 500 points." Here, a very typical and serious factual illusion occurs: "Wrath of the Abyss" is misjudged as "Ice Staff" (attribute illusion), and 2500 health points are seriously misjudged as 500 points (numerical illusion).

[0024] 103. Perform structured data extraction on the multimodal data to obtain structured objective fact data; If we attempt to rely solely on one large model to correct another, we inevitably fall into a logical deadlock of "using uncertainty to correct uncertainty," because the outputs of large models are essentially probability distributions rather than deterministic facts. Therefore, this embodiment returns to the classic expert system concept, utilizing small discriminative models that have been highly specialized and perform exceptionally well on a single vertical task to obtain an absolute factual benchmark. These small models lack the ability to generate lengthy narratives, but they possess extremely high accuracy on specific tasks, with very low inference latency and minimal resource consumption. By allowing these independent expert models to work in parallel, multi-dimensional and comprehensive fact extraction can be achieved, completely eliminating the systematic biases that may exist with a single model.

[0025] This step employs three deeply decoupled core expert models, each with its own independent inference engine and GPU memory pool: (1) Text Recognition Model: Used to extract text information from multimodal data. In game scenarios, UI text often has strong glowing effects, anti-aliasing, semi-transparent background overlays, and even textures on 3D curved surfaces. The text recognition model accurately locates and segments text lines through deformable convolution and an attention-based text region detection network, and then converts the pixel array into a precise string through the recognition network. It does not understand the meaning of health points, but is only responsible for translating the pixel shape into the number 2500.

[0026] (2) Object detection model: used to extract key object information from multimodal data. For a specific game, multi-angle 2D materials rendered from the 3D model in the game are used as the training set for domain fine-tuning to make it have pixel-level sensitivity to key objects such as "staff model", "armor name", and "potion type". The model outputs a structured list with bounding box coordinates, category labels and confidence scores.

[0027] (3) Event Sequence Analysis Model: This is a lightweight classification network based on Long Short-Term Memory (LSTM) to extract information on the development stages of events corresponding to multimodal data. It does not rely on a single frame image, but instead receives the image feature stream of a short time window (such as the first 5 seconds). By analyzing the periodic changes in the global brightness of the screen (such as the special effects of thunderstorm weather skills), the macroscopic movement trend of the minimap icon position, and the appearance and disappearance of specific UI elements (such as countdown), it outputs the structured classification label of the current macroscopic stage.

[0028] The three expert models mentioned above run asynchronously and concurrently in independent thread pools. Finally, through a data aggregation gateway, their respective output strings, detection lists, and category labels are uniformly mapped and aggregated into a JSON structured data set (denoted as F). structured That is, objective factual data.

[0029] For example, given the same screenshot filled with special effects and distractions, the text recognition model can penetrate the semi-transparent UI background and accurately extract: {"health_point": "2500", "mana_point": "800", "skill_name": "Meteor"}; the object detection model can ignore fire effects and extract: {"weapon": "Abyssal Fury Arcane Staff", "armor": "Dark Gold Breastplate"} based on specific patterns at the bottom of the staff; the event sequence analysis model can extract: {"phase": "Final Boss Battle"} by analyzing the appearance of specific characters in the background. Finally, these are merged into a JSON structured data set. F structured = {"weapon": "Abyssal Fury Arcane Staff", "health_point": 2500, "mana_point": 800, "armor": "Unique Breastplate", "phase": "Final Boss Battle"}.

[0030] This JSON data represents the undeniable objective and absolute fact of the multimodal data (i.e., the screenshot) at the current time and space point.

[0031] 104. Perform fact discrepancy detection between the preliminary description text and the objective fact data; Through steps 102 and 103 above, two sets of data in completely different semantic spaces were obtained: T, representing the model's subjective and fuzzy cognition. raw (A coherent passage of natural language) and F, representing absolute objectivity and discreteness. structured (A bunch of JSON key-value pairs). Directly comparing them using traditional string matching or regular expressions is completely ineffective. For example, the text might state "health is very high, almost full," while the JSON would show "health_point": 2500 (assuming full health is 3000). There's no literal overlap, but the semantics are consistent. Conversely, the text might state "used a medium-level healing potion," while the JSON would show "item": "basic healing potion." The literal meaning is highly similar, but there's a serious semantic conflict regarding the level of fact. Therefore, a mechanism with deep semantic understanding, logical reasoning, and common-sense judgment capabilities is needed to discover these hidden "conflicts."

[0032] This embodiment introduces a large language model as the "judge." To avoid the large language model itself from generating illusions or diverging, the design of the pre-set detection prompts in this step is an extremely precise engineering process. The core prompts are set as highly binding instructions, such as "You are a strict fact-checking judge. Your only task is to compare 'textual descriptions' with 'structured facts.' If any entity name, quantity, size, or state attribute in the textual description contradicts or does not match the structured facts, output 'Yes'; if the textual description fully conforms to the facts, or if the text does not mention a certain fact (which should not be considered a conflict), output 'No'. Outputting any explanatory text is strictly prohibited; only 'Yes' or 'No' can be output." This design, by depriving the large language model of its "explanatory power," greatly reduces its space for generating illusions. Then, the detection prompts, T... raw and F structured After being concatenated using specific delimiters, the data is fed into a large language model for processing. Within its high-dimensional hidden space, the large language model maps fuzzy entities of natural language to precise key-value pairs in JSON, performing rigorous logical comparisons.

[0033] Suppose the input to the large language model is: "[Text Description]: The player is holding the Frost Staff and preparing to cast a spell. Currently, the player's remaining health is very low, approximately 500 points. [Structured Fact]: {"weapon": "Abyssal Fury Arcane Staff", "health_point": 2500}.

[0034] The implicit internal reasoning process of the large language model is as follows: The "Frost Staff" in the text corresponds to the "weapon" key in JSON. The values ​​"Abyss Fury Arcane Staff" and "Frost Staff" are different items in the game entity library, resulting in an entity conflict; The "approximately 500 points" in the text corresponds to the "health_point" key in JSON. The values ​​2500 and 500 have a huge difference in magnitude, resulting in a numerical conflict.

[0035] Despite the complexity of the conflict, the strong constraint of the cue words ensured that the large language model's final output was firmly locked to "yes." This successfully and accurately captured the hidden factual discrepancies. This cue-driven detection scheme possesses extremely strong cross-domain generalization capabilities, completely eliminating the enormous development costs of rewriting the conflict rule engine for each new game.

[0036] 105. If a factual discrepancy is detected, then a preference sample for fine-tuning the multimodal large language model is constructed based on the original prompt words, the preliminary description text, and the objective factual data.

[0037] Traditional methods of constructing preference samples require manual selection of the better answer from multiple different responses generated by the model, based on subjective feelings. This leads to significant costs and inconsistencies among labelers. In contrast, the direct preference optimization algorithm used in this embodiment only needs to tell the model "which is good and which is bad," thus completely abandoning subjective human standards and instead utilizing irrefutable "objective facts" (i.e., F) from the physical world. structured Using "100% factual description" as the absolute and sole benchmark, it automatically and mechanically defines "descriptions that are 100% factual" as good and "descriptions that contain even the slightest illusion" as bad. This fundamentally eliminates the subjectivity of preference data, providing the purest and sharpest optimization signal for direct preference optimization algorithms.

[0038] (1) Construct negative samples (T) rejected In preference alignment training, negative samples refer to model outputs that are judged as "worse", "do not meet the expected standard", or "not preferred" under the same input conditions. They represent the generation tendencies that the model should suppress and discard.

[0039] This embodiment directly uses the preliminary description text T obtained in step 102. raw These are marked as negative samples without modification. Because T raw The text not only contains factual errors (such as "Frost Staff"), but also unique "phrasing habits," "sentence structures," and "hallucination triggering patterns" specific to the multimodal large language model under the current parameter state. Therefore, manually modifying errors in descriptive text is not recommended. If the phrasing is manually modified, the direct preference optimization algorithm will be unable to accurately "correct" errors during training because it cannot find matching sentence features. Preserving the original errors is crucial to ensuring that subsequent fine-tuning can accurately optimize the model's pathological neurons.

[0040] (2) Constructing positive samples (T) chosen In preference alignment training, positive samples refer to the model outputs that are judged as "better", "more in line with the expected standard", or "more preferred" when faced with the same input conditions (such as the same image and the same prompt word). They represent the generation tendency that the model should learn and strengthen.

[0041] Based on objective factual data F structured Generate positive samples. This embodiment provides two implementation methods with complementary characteristics: Method 1: Template Filling. For specific business scenarios, a large number of predefined string templates with placeholders are used, such as: "In the current screen, the player is equipped with {weapon}, has a health value of {health_point}, and is in the {phase} stage." The system directly parses the values ​​from the JSON and replaces them with the placeholders using a script. The advantage of this method is that the computational cost is close to zero, and the generation latency is in the millisecond range, making it suitable for large-scale deployment in real-time online learning scenarios where the timeliness of data construction is extremely important. The disadvantage is that the generated text is relatively rigid and lacks diversity.

[0042] Method Two: Large Language Model Refinement and Generation Method. This method involves calling a generative large language model with a massive number of parameters and extremely powerful language capabilities, and providing highly stringent customized prompts: "You are a top-notch game commentator. Please generate a vivid, professional, and fluent commentary based strictly on the following objective factual JSON data. Core constraint: You are absolutely not allowed to fabricate any information that does not exist in the JSON (such as unmentioned skills or monsters that have not appeared). You must be 100% unconditionally faithful to the factual data." Upon receiving the instructions, the large language model leverages its powerful lexical organization capabilities to transform the dry, JSON-formatted objective factual data into a high-quality natural language description.

[0043] To facilitate subsequent model fine-tuning, it is necessary to further refine (the original prompt words P, T) chosen , T rejected The data is packaged into a strict triplet structure and persistently stored in a distributed training data pool.

[0044] Assume the original prompt P is "describe player state". Negative sample T rejected The system outputs the message: "The player is holding the Frost Staff and preparing to cast a spell. Their remaining health is very low, approximately 500 points." The system uses a generative large language model refinement method, taking JSON factual data as input and the large language model as the output of a positive sample T. chosen "In this final boss battle, the player, wielding the Abyss Fury Arcane Staff, has abundant mana but only 2500 health points remaining, making the situation dire." The final damage output trio (P, T) chosen , T rejected The data was fed into the data pool. This triplet pinpointed the model's cognitive biases in the two specific dimensions of "staff name" and "health value" with remarkable precision.

[0045] Furthermore, in automated data pipelines, if the large language model outputs "no" (i.e., no factual difference detected) during factual difference detection in step 104, it indicates that the multimodal large language model performed perfectly on this specific sample, not only avoiding illusions but also potentially demonstrating excellent semantic organization capabilities. In traditional one-off data filtering schemes, such "correct but useless statements" are often discarded directly because they cannot provide "contrast differences" for direct preference optimization algorithms. From a machine learning perspective, correct examples also have irreplaceable training value, especially in consolidating the model's basic expressive capabilities and preventing "capability degradation" during subsequent correction processes.

[0046] In an alternative embodiment, if no difference is detected, then T is determined. raw For high-quality text, it is paired with the original prompt word P and converted into a data format conforming to the supervised fine-tuning standard: {"instruction": P, "output": T_{raw}}. The system maintains two isolated cache queues: one for storing the preference triples produced in step 105, and the other for storing the supervised fine-tuning samples produced in this step. When the data is written to disk, it is written to different distributed file system paths.

[0047] Assuming a simple game screen is input, the multimodal large language model outputs T. raw "The player is standing in the center of the safe zone, without any special items." The large language model compares this data with the factual JSON and outputs "No." The system then converts this data pair into supervised fine-tuning samples for storage. In the final model training phase, a mature industry-standard combined training strategy of "supervised fine-tuning warm-up + preference alignment" can be adopted: first, use these undifferentiated supervised fine-tuning samples to perform several training batches of supervised fine-tuning on the model, solidifying its basic language generation baseline; then switch to preference samples to focus on eliminating its tendency to produce illusions. This data flow mechanism achieves extremely high data utilization.

[0048] This embodiment cleverly utilizes the conflict between the model's own erroneous output (illusion) and external multi-source objective facts to automatically generate high-quality negative and positive samples. This completely eliminates the expensive manual preference labeling process, establishing a fully automated pipeline for fine-tuning sample production and significantly reducing model alignment costs. The construction benchmark for preference samples comes from structured facts extracted by expert models such as text recognition and object detection, rather than human subjective judgment. This ensures that the positive samples in the constructed preference samples have absolute factual correctness and can be used to accurately eliminate factual illusions in the model.

[0049] Please see Figure 2 One embodiment of the multimodal large language model fine-tuning method in this invention includes: 201. Obtain multimodal data samples and preference samples, wherein the preference samples include original prompt words, positive samples and negative samples, and the preference samples are obtained using the sample construction method described above; Model fine-tuning involves performing two forward propagation calculations on the same set of multimodal data with two different text segments (positive and negative). Improper data loading logic can lead to excessive redundant GPU memory usage and a significant waste of computational resources. Furthermore, it is absolutely essential to ensure that the obtained preference samples (P, T) are accurate. chosen , T rejected The image tensor I and its corresponding multimodal data samples are strictly aligned within a batch. If misalignment occurs, the model will learn a completely reversed optimization direction, causing training to fail entirely. Efficient loading can be achieved by defining a complex data loading function that inherits from the Dataset class.

[0050] When the data loading function retrieves a sample by index, it simultaneously reads an image from the disk and performs the same preprocessing (scaling, normalization) as in step 101 to obtain the image tensor I; then it reads the corresponding preference sample. For text processing, a word segmenter perfectly matching the model to be fine-tuned is used to segment the original prompt word P and the positive sample T respectively. chosen The concatenated string, and P and the negative sample T rejected The concatenated string is segmented and a special terminator is added. To accommodate matrix operations on the GPU, all sequences must be strictly padded to the maximum length within the current training batch, and a corresponding attention mask matrix is ​​generated to filter out meaningless padding characters during computation.

[0051] Suppose we have a batch of data with a batch size of 2. Sample 1: The image is "a mage casting a fireball", P is the "description of the action", T... chosen "The mage cast a fireball," T rejected For example, "The mage cast a freeze spell." Sample 2 is similar. This data is converted in memory into an image tensor of shape [2, 3, 448, 448] and two integer tensors of shape [2, Max_Seq_Len].

[0052] 202. Input the original prompt words, the positive samples, the negative samples, and the multimodal data samples into the multimodal large language model to be fine-tuned, and output the first probability of generating the positive samples and the second probability of generating the negative samples; The multimodal data sample image tensor I first extracts image features through a visual encoder and then fuses them with the text features of the original cue word P to obtain a unified sequence of hidden state vectors containing both visual and instructional information (denoted as conditional input X). This input condition is an absolutely fair starting point. Subsequently, the model must perform two independent forward propagations under the same conditional input X: First forward propagation: Connect X with the forward sample T chosen The text features are concatenated and input into the decoding layer of the language model. The model autoregressively calculates the probability of generating each token in Tchosen given X, and sums these probabilities after taking the logarithm, to obtain the first probability.

[0053] Second forward propagation: Connect X with the negative sample T rejected The text features are concatenated, and similarly, the generated T is calculated in the decoding layer. rejected The second probability is obtained from the logarithmic probability of the entire sequence.

[0054] For example, the input condition X is [image features of a mage waving a staff + "description of the action"]. The model calculates the log probability of generating "The mage cast a fireball" (positive), assuming the first probability is -3.0 (because the model may not have learned this word well before). The model calculates the log probability of generating "The mage cast a freeze spell" (negative, i.e., illusion), assuming the second probability is -1.5 (indicating that under the current untuned model parameters, due to corpus bias, it is more inclined to output erroneous illusionary text, because -1.5 > -3.0). These two values ​​truly reflect the model's current distorted cognitive state.

[0055] 203. Based on the difference between the first probability and the second probability, construct a loss function, and based on the loss function, update the parameters of the multimodal large language model through backpropagation.

[0056] This embodiment employs the Direct Preference Optimization (DPO) algorithm to train a multimodal large language model for preference alignment. DPO is an offline training algorithm oriented towards preference alignment, used to directly train preference pairs (P, T) from the data. chosen ,T rejected Fine-tune the multimodal large language model to better align it with user-defined or "objective fact" standards of good / bad. Preference samples are triplet data (P, T). chosen , T rejected ), where P is the original prompt word, T chosen The answer is "preferred / better", T rejected The answer is "not preferred / worse".

[0057] The direct preference optimization algorithm constructs a loss function using the difference between the two probabilities obtained in step 202. This loss function aims to maximize the model's preference on T. chosen The generation probability of T is minimized at the same time. rejected The generation probability. During backpropagation, the partial derivatives of the loss function with respect to each parameter of the model are automatically calculated for updating the model parameters.

[0058] For example, suppose the current probability difference is negative (-3.0 - (-1.5) = -1.5). The calculated loss function value will then be a relatively large positive value (e.g., 0.8). After gradient backpropagation, parameter updates begin. This update is extremely directional: it forcibly increases the weights of the feature corresponding to the "fireball red effect" in the visual encoder, and forcibly increases the connection weights of the Chinese character tokens "fire" and "ball" in the language model when this visual feature is seen; simultaneously, it severely penalizes the weights of the output "ice" and "frozen".

[0059] After thousands of similar triples (including various error corrections such as staff names and health values), in future inference, when a similar game scene is input again, the first probability will soar to -0.5, and the second probability will plummet to -8.0, with the difference becoming a positive 7.5. At this point, the loss function value approaches 0.

[0060] This embodiment uses the constructed preference samples to directly train preference optimization, thereby adjusting the model's own parameter weights so that it learns to focus on objective facts and actively reduces illusions in future reasoning, thus achieving the self-evolution of the model's capabilities.

[0061] The sample construction method in the embodiments of the present invention has been described above. The sample construction apparatus in the embodiments of the present invention will be described below. Please refer to [link / reference]. Figure 3 One embodiment of the sample construction apparatus in this invention includes: Module 301 is used to acquire multimodal data and original prompt words; The generation module 302 is used to perform semantic understanding on the multimodal data based on the original prompt words to obtain the preliminary description text corresponding to the multimodal data; Extraction module 303 is used to extract structured data from the multimodal data to obtain structured objective fact data; Detection module 304 is used to perform fact difference detection between the preliminary description text and the objective fact data; The construction module 305 is used to construct preference samples for preference alignment training of a multimodal large language model based on the original prompt words, the preliminary description text, and the objective fact data if a factual discrepancy is detected.

[0062] In one optional embodiment, the expert model includes a text recognition model, an object detection model, and an event sequence analysis model; the text recognition model is used to extract text information from the multimodal data, the object detection model is used to extract key object information from the multimodal data, and the event sequence analysis model is used to extract event progression stage information corresponding to the multimodal data.

[0063] In an optional embodiment, the detection module 304 is specifically used for: Obtain preset detection prompts for fact discrepancy detection; The detection prompts, the preliminary description text, and the objective fact data are input into a pre-set large language model to perform fact discrepancy detection, and the fact discrepancy detection results are output.

[0064] In an optional embodiment, the construction module 305 is specifically used for: If a factual discrepancy is detected, the preliminary description text is used as a negative sample representing the error generation tendency of the multimodal large language model, and a positive sample representing the correct generation tendency of the multimodal large language model is generated based on the objective factual data. The original prompt words, the positive samples, and the corresponding negative samples are combined to obtain triplet preference samples for fine-tuning the multimodal large language model.

[0065] In an optional embodiment, the construction module 305 is further configured to: Mapping the objective factual data to a preset text description template yields positive samples representing the correct generation tendency of the multimodal large language model; or By using customized prompt words to instruct the generative large language model to generate factual description text corresponding to the objective factual data, the factual description text is used as a positive sample representing the correct generation tendency of the multimodal large language model.

[0066] In an optional embodiment, the construction module 305 is further configured to; If no factual discrepancies are detected, the preliminary description text is determined to be high-quality text, and the preliminary description text and the original prompt words are used as samples for model-supervised fine-tuning.

[0067] Since the embodiments of the device part correspond to the embodiments of the above method, the description of the sample construction device provided by the present invention should refer to the above method embodiments. The present invention will not be described again here, but it has the same beneficial effects as the above sample construction method.

[0068] above Figure 3The sample construction device in the embodiments of the present invention will be described in detail from the perspective of modular functional entities. The computer device in the embodiments of the present invention will be described in detail from the perspective of hardware processing.

[0069] Figure 4 This is a schematic diagram of the structure of a computer device 500 provided in an embodiment of the present invention. The computer device 500 can vary significantly due to different configurations or performance characteristics. It may include one or more central processing units (CPUs) 510 (e.g., one or more processors) and a memory 520, and one or more storage media 530 (e.g., one or more mass storage devices) for storing application programs 533 or data 532. The memory 520 and storage media 530 can be temporary or persistent storage. The program stored in the storage media 530 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the computer device 500. Furthermore, the processor 510 may be configured to communicate with the storage media 530 and execute the series of instruction operations in the storage media 530 on the computer device 500.

[0070] Computer device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input / output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 4 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0071] The present invention also provides a computer device, the computer device including a memory and a processor, wherein the memory stores computer-readable instructions, and when executed by the processor, the computer-readable instructions cause the processor to perform the steps of the sample construction method and / or the steps of the multimodal large language model fine-tuning method in the above embodiments. The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the computer causes the computer to perform the steps of the sample construction method and / or the steps of the multimodal large language model fine-tuning method.

[0072] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0073] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0074] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A sample construction method, characterized in that, The sample construction method includes: Acquire multimodal data and raw prompt words; Based on the original prompt words, semantic understanding is performed on the multimodal data to obtain the preliminary descriptive text corresponding to the multimodal data; Structured data extraction is performed on the multimodal data to obtain structured objective fact data; Perform fact discrepancy detection between the preliminary description text and the objective fact data; If a factual discrepancy is detected, then a preference sample for fine-tuning the multimodal large language model is constructed based on the original prompt words, the preliminary description text, and the objective factual data.

2. The sample construction method according to claim 1, characterized in that, The objective factual data includes textual information, key object information, and information on the stage of event progression in the multimodal data.

3. The sample construction method according to claim 1, characterized in that, The step of performing fact discrepancy detection between the preliminary description text and the objective fact data includes: Obtain preset detection prompts for fact discrepancy detection; The detection prompts, the preliminary description text, and the objective fact data are input into a pre-set large language model to perform fact discrepancy detection, and the fact discrepancy detection results are output.

4. The sample construction method according to claim 1, characterized in that, If a factual discrepancy is detected, the preference sample constructed for fine-tuning the multimodal large language model based on the original prompt words, the preliminary description text, and the objective factual data includes: If a factual discrepancy is detected, the preliminary description text is used as a negative sample representing the error generation tendency of the multimodal large language model, and a positive sample representing the correct generation tendency of the multimodal large language model is generated based on the objective factual data. The original prompt words, the positive samples, and the corresponding negative samples are combined to obtain triplet preference samples for fine-tuning the multimodal large language model.

5. The sample construction method according to claim 4, characterized in that, The generation of positive samples representing correct generation tendencies based on the aforementioned objective factual data includes: Mapping the objective factual data to a preset text description template yields positive samples representing the correct generation tendency of the multimodal large language model; or By using customized prompt words to instruct the generative large language model to generate factual description text corresponding to the objective factual data, the factual description text is used as a positive sample representing the correct generation tendency of the multimodal large language model.

6. The sample construction method according to any one of claims 1-5, characterized in that, The sample construction method further includes: If no factual discrepancies are detected, the preliminary description text is determined to be high-quality text, and the preliminary description text and the original prompt words are used as samples for model-supervised fine-tuning.

7. A method for fine-tuning a multimodal large language model, characterized in that, The multimodal large language model fine-tuning method includes: Obtain multimodal data samples and preference samples, wherein the preference samples include original prompt words, positive samples, and negative samples, and the preference samples are obtained using the sample construction method as described in any one of claims 1-6; The original prompt words, the positive samples, the negative samples, and the multimodal data samples are input into the multimodal large language model to be fine-tuned, and the first probability of generating the positive samples and the second probability of generating the negative samples are output. Based on the difference between the first probability and the second probability, a loss function is constructed, and based on the loss function, the parameters of the multimodal large language model are updated through backpropagation.

8. A sample construction apparatus, characterized in that, The sample construction device includes: The acquisition module is used to acquire multimodal data and raw prompt words; The generation module is used to perform semantic understanding on the multimodal data based on the original prompt words to obtain preliminary description text corresponding to the multimodal data; The extraction module is used to extract structured data from the multimodal data to obtain structured objective fact data; The detection module is used to perform fact difference detection between the preliminary description text and the objective fact data; A construction module is used to construct preference samples for fine-tuning the multimodal large language model based on the original prompt words, the preliminary description text, and the objective fact data if a factual discrepancy is detected.

9. A computer device, characterized in that, The computer device includes: a memory and at least one processor, wherein the memory stores instructions; the at least one processor invokes the instructions in the memory to cause the computer device to perform the sample construction method as described in any one of claims 1-6, and / or the multimodal large language model fine-tuning method as described in claim 7.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the sample construction method as described in any one of claims 1-6, and / or the multimodal large language model fine-tuning method as described in claim 7.

Images