A method, system, device, and medium for fine-tuning a multimodal large language model

Abstract

This application relates to a method, system, device, and medium for fine-tuning a multimodal large language model, belonging to the interdisciplinary field of artificial intelligence and computer vision. The fine-tuning method includes: acquiring original business scene images and preprocessing them to obtain preprocessed images; using annotation tools to annotate bounding box coordinates and define semantic labels for entity targets in the preprocessed images, outputting a structured annotation file; constructing a training sample set including multi-turn text-image dialogue based on the preprocessed images and the structured annotation file; loading the pre-trained multimodal large language model, configuring low-rank matrix factorization parameters, and generating a fine-tuning instruction set; inputting the training sample set into the pre-trained multimodal large language model, performing joint training operations based on the fine-tuning instruction set, and outputting the fine-tuned multimodal large language model. This application improves the recognition accuracy, interaction capabilities, and system usability of visual question answering systems in practical application scenarios.

Description

Technical Field

[0001] This application relates to the field of interdisciplinary technology of artificial intelligence and computer vision, and in particular to a method, system, device and medium for fine-tuning a multimodal large language model for enhancing a visual question answering system. Background Technology

[0002] In recent years, multimodal large language models based on the Transformer architecture (such as BLIP-2 and Qwen-VL) have demonstrated strong generalization capabilities in open-domain image-text question answering tasks. These models, pre-trained on massive amounts of internet image and text data, establish cross-modal mappings from pixels to semantics, enabling natural language descriptions or simple inferences about image content. However, when these general-purpose models are applied to vertical business systems (such as smart TV navigation and educational app interfaces), their technical limitations become increasingly apparent.

[0003] In smart TV navigation systems, the interface typically contains densely packed functional areas (such as "Live," "On-Demand," "VIP Zone," and "Games"), each carrying specific business semantics and adhering to strict spatial arrangement rules. While existing general-purpose visual question-answering systems can recognize text or icon elements on the interface, they struggle to distinguish between adjacent controls with similar visual features but vastly different business meanings, and are even less able to return the precise geometric location of the target within the interface. This lack of granular recognition leads to a persistently high error rate in navigation command execution; for example, the voice command "Open VIP Zone" might mistakenly trigger a regular video entry.

[0004] Furthermore, while responses generated by general visual question-answering systems using natural language (such as "the VIP button is in the upper right corner of the screen") conform to human reading habits, they cannot be automatically parsed and utilized by downstream business systems. Development teams need to design complex regular expressions or keyword matching rules to extract effective information, which not only increases system maintenance costs but also leads to navigation errors due to the ambiguity of natural language (such as the vague definition of the coordinate range of "upper right corner"). In addition, existing technologies lack support for continuous interaction scenarios. When users ask questions consecutively, the general model, lacking a dialogue state tracking mechanism, can only respond to each round of questions in isolation, forcing users to repeatedly describe contextual information, resulting in low interaction efficiency. Summary of the Invention

[0005] To improve the recognition accuracy, interactivity, and usability of visual question answering systems in practical application scenarios, this application provides a method, system, device, and medium for fine-tuning a multimodal large language model.

[0006] Firstly, this application provides a method for fine-tuning a multimodal large language model, employing the following technical solution:

[0007] A method for fine-tuning a multimodal large language model, the fine-tuning method comprising:

[0008] The original business scene image is acquired and preprocessed to obtain the preprocessed image;

[0009] The bounding box coordinates and semantic labels of entity targets in the preprocessed image are annotated using annotation tools, and a structured annotation file is output.

[0010] Based on the preprocessed images and structured annotation files, a training sample set including multi-turn text-to-image dialogues is constructed;

[0011] Load a pre-trained multimodal large language model, configure low-rank matrix factorization parameters, and generate a fine-tuning instruction set;

[0012] The training sample set is input into the pre-trained multimodal large language model, and joint training is performed based on the fine-tuning instruction set to output the fine-tuned multimodal large language model.

[0013] By adopting the above technical solutions, a systematic design based on structured labeled data construction, efficient low-rank parameter fine-tuning, and multi-task joint optimization has been implemented, achieving accurate adaptation and efficient deployment of multimodal large language models in vertical business scenarios. First, the joint annotation mechanism of bounding box coordinates and semantic labels enables the model to break through the limitations of traditional natural language output, forming machine-parseable structured responses that directly support the automated processing needs of business systems. Second, the low-rank matrix factorization strategy, while preserving pre-training knowledge, completes model tuning with extremely low parameter update volume, overcoming the industry pain point of high training resource consumption for multimodal models. Finally, the introduction of multi-turn dialogue training data and a context-aware mechanism significantly enhances the model's focus tracking capability in continuous interaction scenarios, providing an efficient and reliable technical path for the industry deployment of visual question answering systems.

[0014] Optionally, the step of constructing a training sample set including multi-turn text-to-image dialogue based on the preprocessed images and structured annotation files includes:

[0015] Parse the semantic tags and corresponding bounding box coordinate data in the structured annotation file to generate a dictionary mapping the entity semantic tag set to the bounding box coordinate values;

[0016] Based on the set of entity semantic tags in the mapping dictionary, generate the initial question text and the initial response text containing structured identifiers;

[0017] Based on at least one entity semantic tag in the initial response text, generate subsequent question text that references the entity semantic tag;

[0018] Based on the mapping dictionary, the bounding box coordinates of the semantic tags of the referenced entities are retrieved, and subsequent response text containing the corresponding coordinate identifiers is generated;

[0019] The initial question text, initial response text, subsequent question text, and subsequent response text are combined into a continuous dialogue sequence;

[0020] The storage paths of the continuous dialogue sequences, preprocessed images, and mapping relationship dictionaries are bound together as a single training sample;

[0021] Aggregate all training samples and encode them line by line into a JSON file to generate a training sample set for multi-turn text-to-image dialogue.

[0022] By adopting the above technical solution and manually designing question-and-answer chains and focus entity referencing mechanisms, the model is forced to learn cross-round visual language associations. Identifiers are used as explicit separators, enabling the model output to adapt to the parsing of the business system. Simultaneously, a mapping dictionary is bound to each sample as labeled truth values, eliminating noise interference and improving fine-tuning accuracy.

[0023] Optionally, the step of inputting the training sample set into a pre-trained multimodal large language model and performing joint training based on the fine-tuning instruction set includes:

[0024] Convolutional encoding is performed on the preprocessed images in the training sample set to generate visual feature vectors;

[0025] Word embedding encoding is performed on the multi-turn text-image dialogue text in the training sample set to generate text feature vectors;

[0026] The visual feature vector and the text feature vector are input into the cross-attention layer, and the fused feature tensor is output.

[0027] Based on the bounding box coordinates in the structured labeled text, spatial coordinate regression is performed on the fused feature tensor to calculate the bounding box loss value;

[0028] Based on the semantic tags in the structured annotation file, entity classification is performed on the fused feature tensor, and the classification loss value is calculated.

[0029] The bounding box loss value and the classification loss value are superimposed to generate a joint loss function;

[0030] According to the freeze instruction in the fine-tuning instruction set, all weight parameters of the multimodal large language model are locked;

[0031] The weight increment is updated based on the low-rank matrix factorization parameters in the fine-tuning instruction set.

[0032] The low-rank matrix factorization parameters are optimized through backpropagation until the joint loss function meets the preset convergence condition or reaches the preset training rounds, at which point the finely tuned multimodal large language model is output.

[0033] By adopting the above technical solutions, the visual encoding layer parses complex images such as navigation interfaces into structured feature vectors, laying the foundation for fine-grained recognition; the text embedding layer guides the model to generate machine-parseable response formats through special encoding of identifiers; the cross-attention mechanism establishes image-text associations, which can support focus tracking in multi-turn dialogues; the joint loss function simultaneously optimizes positioning accuracy and classification accuracy, ensuring that the output has both semantic labels and coordinate information, while the LoRA freeze-increment update strategy significantly reduces the resource threshold.

[0034] Optionally, spatial coordinate regression is performed on the fused feature tensor, and the bounding box loss value is calculated using the SmoothL1 loss function; entity classification is performed on the fused feature tensor, and the classification loss value is calculated using the cross-entropy loss function.

[0035] Optionally, after outputting the fine-tuned multimodal large language model, the following steps may also be included:

[0036] The fine-tuned multimodal large language model is encapsulated into a callable service interface;

[0037] The input protocol of the service interface is defined as receiving a request message containing image data and text questions, and the output protocol is returning structured response data containing at least one entity label and its corresponding bounding box coordinates.

[0038] Receive a request message sent by a user client and parse the session identifier in the request message;

[0039] Determine whether the session identifier is null; if not, retrieve the dialogue state machine bound to the session identifier from the server's in-memory database.

[0040] Historical data is obtained by retrieving stored historical entity labels and bounding box coordinates from the dialogue state machine.

[0041] The historical data is combined with the text question in the current request message to form context-enhanced input text;

[0042] The input text and the decoded image data from the current request message are input together into the fine-tuned multimodal large language model to generate new entity labels and corresponding bounding box coordinates, thus obtaining new data;

[0043] The newly added data is merged with the historical data and then encapsulated according to a preset structured data format to generate a final response message, which is then returned to the user client.

[0044] The dialogue state machine bound to the session identifier is updated based on the new data and stored in the in-memory database.

[0045] By adopting the above technical solutions, a complete technology chain for a conversation-aware visual question-answering system was constructed. By combining conversation identifiers with dialogue state machines, the interaction limitations of traditional single-turn question-answering were broken, enabling the model to have context tracking capabilities. Furthermore, the design of structured input and output protocols enabled lossless conversion from raw data to business-parseable formats, while the dynamic context enhancement mechanism significantly improved target localization accuracy in complex scenarios through spatiotemporal semantic fusion.

[0046] Optionally, the step of combining the historical data with the text question in the current request message into context-enhanced input text includes:

[0047] Extract the historical entity labels and corresponding bounding box coordinates recorded during the three most recent interactions from the historical data;

[0048] The historical entity labels, bounding box coordinates, and recording time are concatenated to form historical context prompt text;

[0049] Concatenate the historical context prompt text with the text question in the current request message to form a continuous text sequence;

[0050] Add a system role identifier at the beginning of the continuous text sequence to generate enhanced input text that conforms to the multi-turn dialogue input format.

[0051] By adopting the above technical solutions, the model can establish spatial evolution logic based on historical coordinate data and accurately locate the target, greatly improving the response accuracy of complex interactive scenarios and providing an engineerable contextual solution for multimodal dialogue systems.

[0052] Optionally, after determining that the session identifier is null, the method further includes:

[0053] Treat the current request message as the first request and generate a unique session identifier;

[0054] Create a new dialogue state machine on the server and initialize it to an empty set;

[0055] The text question in the current request message is used as input text to decode the current request message and obtain the current image data.

[0056] The input text and the current image data are input together into the fine-tuned multimodal large language model to generate an intermediate output result containing the newly added entity labels and their corresponding bounding box coordinates;

[0057] The intermediate output is encapsulated into a structured response message and returned to the user client, with the unique session identifier carried in the response message;

[0058] The dialogue state machine is updated based on the intermediate output results, and the updated dialogue state machine is bound to the unique session identifier and stored in the memory database.

[0059] By adopting the above technical solutions, from ensuring the global uniqueness of session identifiers and the pure initialization of state machines to the accurate alignment and joint reasoning of multimodal data, the engineering encapsulation and state persistence of structured output are finally completed.

[0060] Secondly, this application provides a multimodal large language model fine-tuning system, which adopts the following technical solution:

[0061] A multimodal large language model fine-tuning system, the fine-tuning system comprising:

[0062] The preprocessing module is used to acquire the original business scene image and perform preprocessing to obtain the preprocessed image;

[0063] The structured annotation module is used to annotate the bounding box coordinates and define semantic labels for entity targets in the preprocessed image using annotation tools, and output a structured annotation file;

[0064] The training sample set construction module is used to construct a training sample set including multi-turn text-to-image dialogue based on the preprocessed images and structured annotation files;

[0065] The model loading module is used to load pre-trained multimodal large language models;

[0066] The fine-tuning instruction generation module is used to configure low-rank matrix decomposition parameters and generate a fine-tuning instruction set.

[0067] The model training module is used to input the training sample set into the pre-trained multimodal large language model, perform joint training operations based on the fine-tuning instruction set, and output the fine-tuned multimodal large language model.

[0068] Thirdly, this application provides a computer device, which adopts the following technical solution:

[0069] A computer device includes a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to perform the steps of the method as described in the first aspect.

[0070] Fourthly, this application provides a computer-readable storage medium, which adopts the following technical solution:

[0071] A computer-readable storage medium storing a computer program that can be loaded by a processor and executed as in any of the methods in the first aspect.

[0072] In summary, this application includes at least one of the following beneficial technical effects:

[0073] 1. This application introduces a mechanism for constructing image and text datasets with bounding box annotations and semantic labels, and combines it with a parameter-efficient LoRA fine-tuning strategy to achieve fine-grained recognition and spatial localization of complex interface elements in vertical scenes.

[0074] 2. In smart TV navigation scenarios, after fine-tuning with a dedicated dataset, the model can transform user questions like "How to enter the VIP area" into structured responses with coordinates, allowing the front-end system to automatically parse and trigger navigation logic. Furthermore, by constructing training samples containing multi-turn dialogue states, the model can associate the focus positions in the user's historical interactions (such as continuously asking "Navigation options → Current focus → Details page"), achieving context-aware continuous question-and-answer, thus solving the response fragmentation problem caused by the lack of state tracking in traditional single-turn dialogue models.

[0075] 3. By deploying the model through the Web UI / API interface, this application can be adapted to business systems such as image recognition and navigation positioning, and realize an end-to-end closed-loop application process. Attached Figure Description

[0076] Figure 1 This is a first flowchart illustrating a multimodal large language model fine-tuning method according to one embodiment of this application.

[0077] Figure 2 This is a second flowchart illustrating a multimodal large language model fine-tuning method according to one embodiment of this application.

[0078] Figure 3 This is a schematic diagram of the third process of a multimodal large language model fine-tuning method according to one embodiment of this application.

[0079] Figure 4 This is a schematic diagram of the fourth process of a multimodal large language model fine-tuning method according to one embodiment of this application.

[0080] Figure 5 This is a schematic diagram of the fifth step of a multimodal large language model fine-tuning method according to one embodiment of this application.

[0081] Figure 6 This is a schematic diagram of the sixth process of a multimodal large language model fine-tuning method according to one embodiment of this application. Detailed Implementation

[0082] To make the purpose, technical solution, and advantages of this application clearer, the following description is provided in conjunction with the appendix. Figure 1-6 The present application will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the application.

[0083] First, the technical terms used in this application will be explained:

[0084] Multimodal: refers to a computing model that can simultaneously process different data types (such as visual images and natural language text) to achieve cross-modal information fusion.

[0085] Vision-Language Model: An artificial intelligence model with the ability to jointly understand images and text, supporting image-text reasoning and task generation.

[0086] Fine-tuning: Based on pre-trained models, task-adaptive adjustments are achieved through retraining on domain-specific datasets.

[0087] LoRA (Low-Rank Adaptation): A parameter-efficient fine-tuning method based on low-rank matrix factorization, which reduces computational resource consumption by adjusting subspace parameters.

[0088] Grounding: In multimodal processing, it associates semantic objects with their spatial coordinates within an image.

[0089] Bounding Box: A rectangular set of coordinates used to label a target object, centered at its top-left corner. and the bottom right corner definition.

[0090] LabelImg: An open-source image annotation tool that supports the generation of VOC / YOLO format annotation files.

[0091] This application discloses a method for fine-tuning a multimodal large language model.

[0092] Reference Figure 1 A method for fine-tuning a multimodal large language model, the fine-tuning method includes:

[0093] Step S101: Obtain the original business scene image and perform preprocessing to obtain a preprocessed image;

[0094] The original business scenario images (such as OTT navigation interfaces and screenshots of educational apps) often have resolution differences (e.g., a mixture of 4K images captured by some devices and 720p images captured by low-end cameras), color space shifts (e.g., RGB channel value deviations or YUV encoding differences), and noise interference (e.g., JPEG compression artifacts). Therefore, by standardizing the size (uniformly scaling the image to a fixed resolution, such as 1024×768) and normalizing the channels (scaling the pixel values ​​to the [0,1] range), the impact of hardware acquisition differences on model training can be eliminated, ensuring the consistency of the input data distribution during the visual feature extraction stage.

[0095] For example, in smart TV navigation scenarios, if the input image is not standardized in size, the model may confuse the relative positions of interface elements due to differences in interface layout across different devices, leading to errors in subsequent annotation and positioning. This step directly affects the accuracy of subsequent annotation data and the model's ability to understand the spatial relationships of the target.

[0096] Step S102: Use the annotation tool to annotate the bounding box coordinates and define semantic labels for the entity targets in the preprocessed image, and output a structured annotation file;

[0097] The core of this step lies in establishing a spatial-semantic joint mapping mechanism between visual entities and semantic concepts. This involves using annotation tools (such as LabelImg) to annotate bounding boxes (formatted as top-left corner coordinates). and the coordinates of the bottom right corner This allows the model to establish a geometric mapping from image pixel space to semantic concepts. For example, in an OTT navigation interface, the annotation tool labels the coordinates of the "VIP Zone" button as (664, 174, 1390, 282) and assigns it the semantic label "VIP". Here, the bounding box defines the absolute position of the target in the image, while the semantic label gives it business meaning. Furthermore, the output structured annotation file not only contains coordinates and labels but also the hierarchical relationships between objects (such as navigation bar → sub-menu items), which is crucial for the multimodal model to understand the interface hierarchy.

[0098] Understandably, explicit spatial annotations using annotation tools address the lack of grounding capabilities in general models. Furthermore, the definition of business semantic tags (such as "children" and "live streaming") enables models to build fine-grained classification systems tailored to specific industry needs, rather than relying on general categories (such as "icons" and "text").

[0099] Step S103: Based on the preprocessed images and structured annotation files, construct a training sample set including multi-turn text-to-image dialogue;

[0100] Specifically, the underlying logic of multi-turn dialogue construction is context state modeling and enhanced intent continuity. Each training sample includes a user question (such as "What are the classifications of the navigation bar?"), an assistant response (such as "It includes <TV><VIP Zone>"), an image path, and an annotation object dictionary. The key design lies in the temporal correlation of dialogue turns. For example, in the first round of Q&A, the user asks "What is the current focus position", and the model needs to combine the "VIP Zone" label mentioned in the previous dialogue and return the corresponding coordinate box in the response. In technical implementation, multi-turn dialogues in the sample set simulate real user interaction scenarios through a state machine mechanism to ensure the reasonableness of context association.

[0101] Among them, the "messages" field of each sample adopts a role-playing structure (alternating between "user" and "assistant"), and the image and annotation objects are bound to the dialogue context, enabling the model to learn to maintain tracking of the image focus area during continuous interaction. This data construction method directly overcomes the "dialogue isolation" defect existing in traditional single-turn Q&A datasets, forcing the model to learn cross-turn visual-language joint reasoning capabilities during training. For example, according to the instruction of "Switch to the game area" in the previous dialogue, the position of the game label in the image can be automatically associated in subsequent Q&A.

[0102] Step S104, load a pre-trained multi-modal large language model, configure low-rank matrix factorization parameters, and generate a fine-tuning instruction set;

[0103] Among them, the fine-tuning instruction set is used to indicate that only the low-rank subspace parameters after weight matrix factorization are updated during the model training process. The principle of Low-Rank Adaptation (LoRA) is based on weight increment parameterization. Assuming the original model weight matrix is , by introducing two low-rank matrices and (where ), the weight update amount is expressed as . When configuring parameters, set the rank dimension r = 8 and the scaling factor α = 32, so that only about 0.1% of the parameters need to be optimized during the training process (for example, only 7 million parameters are updated among the 7 billion parameters of the original model).

[0104] Specifically, the specific functions of the fine-tuning instruction set include: (1) freezing all parameters of the original model to prevent catastrophic forgetting caused by full-parameter fine-tuning; (2) specifying which layers to apply low-rank adaptation (usually selecting the query / key matrix of the attention layer); (3) defining optimizer hyperparameters (such as learning rate, weight decay). For example, in the Qwen2.5-VL-7B model, LoRA only acts on the cross-attention module of the visual encoder to ensure efficient adjustment during the visual-language modality alignment process, while the text decoder maintains the original weights.

[0105] Understandably, the significance of this step lies in significantly reducing GPU memory usage (from 80GB for full fine-tuning to below 24GB) through the Parameter-Efficient Fine-Tuning (PEFT) strategy, enabling the model to be trained on consumer-grade GPUs while retaining the general knowledge accumulated during the pre-training phase.

[0106] Step S105: Input the training sample set into the pre-trained multimodal large language model, perform joint training operation based on the fine-tuning instruction set, and output the fine-tuned multimodal large language model.

[0107] The final fine-tuned model output possesses visual positioning capabilities and structured output capabilities adapted to business scenarios. For example, given an input image of a smart TV interface and the question "How to enter the VIP zone?", the model will return a JSON-formatted response {"objects":[{"label":"VIP zone","bbox":[664,174,1390,282]}]}, which downstream systems can automatically parse and execute the redirection operation.

[0108] In the above implementation, a systematic design based on structured labeled data construction, efficient low-rank parameter fine-tuning, and multi-task joint optimization achieves accurate adaptation and efficient deployment of multimodal large language models in vertical business scenarios. First, the joint annotation mechanism of bounding box coordinates and semantic labels enables the model to break through the limitations of traditional natural language output, forming machine-parseable structured responses that directly support the automated processing needs of business systems. Second, the low-rank matrix factorization strategy, while preserving pre-training knowledge, completes model tuning with extremely low parameter update volume, overcoming the industry pain point of high training resource consumption for multimodal models. Finally, the introduction of multi-turn dialogue training data and a context-aware mechanism significantly enhances the model's focus tracking capability in continuous interaction scenarios, providing an efficient and reliable technical path for the industry-wide deployment of visual question answering systems.

[0109] Reference Figure 2 As one implementation of step S103, the step of constructing a training sample set including multi-turn text-to-image dialogue based on preprocessed images and structured annotation files includes:

[0110] Step S201: Parse the semantic tags and corresponding bounding box coordinate data in the structured annotation file to generate a dictionary of mapping relationships between entity semantic tag sets and bounding box coordinate values;

[0111] Structured annotation files (such as JSON format) are the product of manual annotation of preprocessed images using tools such as LabelImg. Essentially, they establish a digital mapping between image semantics and spatial location. Each annotation object contains two key data: semantic labels (such as "VIP Zone"), which define the business attributes of the target entity in the image (distinguished from general categories, such as "video"); and bounding box coordinates (in the format (x1, y1, x2, y2)), which precisely describe the physical location of the target in the image using pixel coordinates.

[0112] Specifically, the parsing process generates a dictionary mapping entity semantic label sets to bounding box coordinate values ​​by reading JSON key-value pairs. This dictionary is stored in the computer as a hash table structure, realizing a label-coordinate lookup function with O(1) complexity, providing a data foundation for the subsequent generation of structured response text. For example, the labeled file {"VIP Zone":[664,174,1390,282]} is parsed into an in-memory dictionary, and the key "VIP Zone" is directly mapped to the coordinate list [664,174,1390,282].

[0113] Step S202: Generate the initial question text and the initial response text containing structured identifiers based on the entity semantic tag set in the mapping relationship dictionary;

[0114] The initial response text embeds entity semantic tags. and Embed the corresponding bounding box coordinates between the identifiers. and Between identifiers;

[0115] Specifically, based on the semantic label set of the mapping dictionary, training data that conforms to the characteristics of visual question answering tasks needs to be constructed. The question generation logic is as follows: construct an initial question based on the business scenario requirements (such as "What are the categories of the navigation bar?"), which needs to guide the model to identify all key entities in the image.

[0116] For example, the structured design of the response text is as follows: Entity semantic tags such as "VIP Zone" are placed in... and Between identifiers, form VIP Zone This design acts as a semantic segmentation marker during model training, forcing the model to learn the ability to recognize the boundaries of labeled text. Coordinate values ​​such as (664, 174, 1390, 282) are placed... and Between, forming (664,174,1390,282) This identifier acts as a coordinate generation trigger, guiding the model to output parsable spatial location data.

[0117] It should be noted that the entity labels are arranged and concatenated according to the interface layout (e.g., from left to right) to ensure that the order of the response text is consistent with the actual image space relationship, thereby enhancing the model's understanding of visual structure.

[0118] Step S203: Based on at least one entity semantic tag in the initial response text, generate subsequent question text that references the entity semantic tag;

[0119] To achieve multi-turn dialogue capabilities, it is necessary to simulate continuous user interaction behavior. At least one entity semantic tag (such as "VIP Zone") can be extracted from the initial response text as the dialogue focus entity. Question construction rule: subsequent question texts must explicitly or implicitly reference this focus entity (such as "Where is the VIP Zone on the screen?"). This design simulates users' follow-up questions to specific goals in real-world scenarios, forcing the model to establish cross-turn entity reference relationships (coreference resolution).

[0120] Understandably, general multimodal models often lose context in multi-turn dialogues due to the lack of explicit focus passing mechanisms. This step ensures that the model maintains attention to the same target in subsequent responses by forcing entity references.

[0121] Step S204: Retrieve the bounding box coordinates of the semantic tags of the referenced entity according to the mapping relationship dictionary, and generate subsequent response text containing the corresponding coordinate identifiers;

[0122] Specifically, based on the name of the focus entity (e.g., "VIP Zone"), the corresponding bounding box coordinates are retrieved from the generated mapping dictionary; these coordinates are then embedded into a preset identifier, forming a structure such as "located in...". (664,174,1390,282) The response text. This step allows the model to learn to dynamically bind semantic labels to coordinate values, while ensuring that all coordinate responses conform to a fixed template that can be parsed by the machine, eliminating the uncertainty of natural language descriptions.

[0123] Step S205: Combine the initial question text, initial response text, subsequent question text, and subsequent response text into a continuous dialogue sequence;

[0124] The initial and subsequent question-and-answer texts are integrated into a message array according to the interaction sequence. This continuous dialogue sequence satisfies the multi-turn dialogue training format of a multimodal large model (such as Qwen-VL).

[0125] Step S206: Bind the continuous dialogue sequence, the storage path of the preprocessed image, and the mapping relationship dictionary into a single training sample;

[0126] Specifically, continuous dialogue sequences provide the linguistic interaction context, preprocessed image paths (such as / data / nav_screen.jpg) are used to associate with the visual input source, and a mapping dictionary serves as the ground truth for label-coordinate supervision. This structure allows the model to access image data, dialogue history, and labeled ground truth simultaneously during training, achieving joint supervised learning of vision, language, and space.

[0127] Step S207: Aggregate all training samples and encode them line by line into a JSON format file to generate a training sample set for multi-round text-to-image dialogue.

[0128] This process involves storing all training sample objects in a list, storing each sample independently as a JSON object row by row, and generating a .jsonl file.

[0129] In the above implementation, by manually designing question-and-answer chains and focus entity referencing mechanisms, the model is forced to learn cross-round visual language associations. Identifiers are used as explicit separators to make the model output adapt to the parsing of the business system. The mapping relationship dictionary is used as labeled truth values ​​and bound to each sample to eliminate noise interference and improve fine-tuning accuracy.

[0130] Reference Figure 3 As one implementation of step S105, the step of inputting the training sample set into the pre-trained multimodal large language model and performing joint training based on the fine-tuning instruction set includes:

[0131] Step S301: Perform convolutional encoding on the preprocessed images in the training sample set to generate visual feature vectors;

[0132] The preprocessed image is input into a convolutional neural network (CNN) or a visual Transformer (ViT), which performs spatial feature extraction through hierarchical convolutional kernels. The convolution operation essentially scans the image by sliding the local receptive field, capturing abstract features from edge textures to semantic objects layer by layer: primary convolutional layers identify basic patterns such as lines and color blocks, while deeper convolutions activate complex structures (such as navigation bar icons and button boundaries). After multiple downsampling operations (such as max pooling), the original pixel matrix is ​​compressed into a high-dimensional visual feature vector, which carries the spatial structural information of the image (e.g., "VIP Zone" in the upper right corner of the screen).

[0133] Step S302: Perform word embedding encoding on the multi-round text-image dialogue text in the training sample set to generate text feature vectors;

[0134] Among them, the multi-turn text-image dialogue text (including user questions and assistant responses) input word embedding layer maps discrete words into dense vectors through a lookup table mechanism.

[0135] Specifically, the specific steps of this process include: (1) Location encoding adds a location sequence vector to each word embedding to preserve the dialogue sequence logic of "What categories are there in the navigation bar?" → "Where is the VIP area?"; (2) Role embedding distinguishes the speaking identities of users and assistants, strengthening the model's cognition of interactive roles; (3) Special encoding of structured identifiers: Human-generated identifiers are assigned independent embedding vectors, which serve as anchor signals to trigger label / coordinate generation. The final output is a sequence of text feature vectors, which are essentially a quantified representation of the semantics of business terms (such as "cloud karaoke").

[0136] Step S303: Input the visual feature vector and the text feature vector into the cross-attention layer and output the fused feature tensor;

[0137] The visual and text feature vectors are input into a multimodal cross-attention layer, which achieves modal interaction through a QKV (Query-Key-Value) mechanism: using text features as the query vector, the model scans the key-value pairs (Key-Value) formed by visual features and calculates the attention weight of the text on the image. For example, when the input question is "What categories are there in the navigation bar?", the model will focus on the feature response of the top area of ​​the image; while the follow-up question "Location of the VIP area" will shift the attention weight to a specific icon area. The weighted visual features and the original text features are residually connected to output a fused feature tensor. This tensor simultaneously encodes the spatial attributes (coordinate trends) of the visual object and the textual semantics (business labels), forming a joint representation.

[0138] Step S304: Based on the bounding box coordinates in the structured labeled text, perform spatial coordinate regression on the fused feature tensor and calculate the bounding box loss value;

[0139] The system integrates a fully connected regression network with fused feature tensors to predict the target bounding box coordinates (x1, y1, x2, y2). Supervision signals are derived from the ground truth coordinates in a structured annotation file (e.g., [664, 174, 1390, 282] annotated with LabelImg). A Smooth L1 Loss function is used to calculate the difference between the predicted and ground truth values: this function enables L2 loss to optimize accuracy when the error is small, and switches to L1 loss to enhance robustness when the error is large.

[0140] The mathematical expression is:

[0141] ;

[0142] In the above formula, Pred i To predict coordinates, gt i These are the actual coordinates.

[0143] Understandably, by forcing the model to learn spatial positioning capabilities, abstract visual features are mapped into specific coordinate outputs, directly solving the problem of "lack of accurate spatial positioning capabilities" in the background technology.

[0144] Step S305: Based on the semantic tags in the structured annotation file, perform entity classification on the fused feature tensor and calculate the classification loss value;

[0145] Specifically, feature tensors are fused and simultaneously input into the classification network to predict entity semantic labels (e.g., "VIP Zone"). Supervision signals originate from a set of semantic labels in labeled files, and cross-entropy loss is used to measure the deviation between the predicted probability distribution and the true labels.

[0146] ;

[0147] in, One-hot encoding of the true value of the category. This represents the probability value output by the softmax function. This loss function helps the model distinguish between fine-grained business tags (such as "watch TV" and "VIP zone"), improving semantic classification accuracy and avoiding the misclassification of heterogeneous targets as a uniform generic class by traditional models.

[0148] Step S306: Overlay the bounding box loss value and the classification loss value to generate the joint loss function;

[0149] Among them, bounding box loss With classification loss A joint loss function is formed by weighting the coefficients:

[0150] .

[0151] Specifically, the weight coefficients α and β are dynamically adjusted according to the importance of the task (default α=1.0, β=0.5). This step achieves multi-task collaborative optimization, providing spatial supervision signals through coordinate regression and enhancing semantic discrimination capabilities through entity classification. The gradients of both are synchronously updated with model parameters through backpropagation. The joint supervision mechanism is the core guarantee for generating structured output (label + coordinates), preventing the model from falling into single-task bias (such as accurate localization but incorrect label), and solving the engineering bottleneck of "unstructured output" at the system level.

[0152] Step S307: Lock all weight parameters of the multimodal large language model according to the freeze instruction in the fine-tuning instruction set;

[0153] Specifically, based on the freeze instruction in the fine-tuning instruction set, all weight parameters of the pre-trained model (Qwen2.5-VL-7B) are set to a non-updatable state. In the computation graph, the property of all weight tensors is set to require_grad=False to prevent gradients from flowing to the original parameters during backpropagation.

[0154] Understandably, this step is a prerequisite for LoRA fine-tuning. Freezing the base model preserves the knowledge it gained during open-domain pre-training (such as general object recognition capabilities), and only adding lightweight modules to adapt to vertical scenes significantly reduces GPU memory usage.

[0155] Step S308: Update the weight increment based on the low-rank matrix factorization parameters in the fine-tuning instruction set;

[0156] Specifically, based on the frozen original model, a trainable low-rank decomposition matrix is ​​injected into the linear layer. The specific operation involves converting the original weight matrix... Decomposed into Among them, the increment From low-rank matrix and Product structure ( ).

[0157] Step S309: Optimize the low-rank matrix factorization parameters through backpropagation until the joint loss function meets the preset convergence condition or reaches the preset training rounds, and output the fine-tuned multimodal large language model.

[0158] Specifically, during training, only the parameters of matrices B and A are optimized: the joint loss function L is calculated via backpropagation. total The gradients of B and A are iteratively updated using an adaptive optimizer (such as AdamW) until the loss converges or the preset number of training epochs is reached. The formula for the adaptive optimizer is:

[0159] .

[0160] In the above implementation, the visual encoding layer parses complex images such as navigation interfaces into structured feature vectors, laying the foundation for fine-grained recognition; the text embedding layer guides the model to generate a machine-parseable response format through special encoding of identifiers; the cross-attention mechanism establishes image-text association, which can support focus tracking in multi-turn dialogues; the joint loss function simultaneously optimizes positioning accuracy and classification accuracy, ensuring that the output has both semantic labels and coordinate information, while the LoRA freeze-increment update strategy significantly reduces the resource threshold.

[0161] Reference Figure 4 As a further implementation of the multimodal large language model fine-tuning method, after the step of outputting the fine-tuned multimodal large language model, the method further includes:

[0162] Step S401: Encapsulate the fine-tuned multimodal large language model into a callable service interface;

[0163] Step S402: Define the input protocol of the service interface as receiving a request message containing image data and text questions, and the output protocol as returning structured response data containing at least one entity label and corresponding bounding box coordinates;

[0164] Specifically, containerization technologies (such as Docker) are used to package model weights, dependency libraries, and inference scripts into independent service units, achieving physical decoupling between model capabilities and the business system. Furthermore, the input protocol of the service interface mandates a bimodal structure for request messages: image data must be encoded as a Base64 string or binary stream, and text requests must be encapsulated in UTF-8 format. This design avoids parsing errors caused by the obfuscation of image and text data in traditional solutions. The output protocol defines a structured response paradigm, requiring the return of at least one entity label and its bounding box coordinates.

[0165] Step S403: Receive the request message sent by the user client and parse the session identifier in the request message;

[0166] Upon receiving a user request message, the system first parses the session identifier in the message header, which serves as the unique carrier of the continuity of the conversation.

[0167] Step S404: Determine if the session identifier is empty; if not, proceed to step S405.

[0168] Step S405: Retrieve the dialogue state machine bound to the session identifier from the server's in-memory database;

[0169] Specifically, a non-empty identifier activates the history context loading mechanism, and the in-memory database retrieves the dialogue state machine bound to that identifier at millisecond speeds via key-value storage. The dialogue state machine is essentially a structured data container, stored in the format {"history_labels":["Watching TV","Games"],"history_boxes":{"Watching TV":[68,184,132,275]}}, which fully records the entities identified in historical interactions and their spatial locations. This step enables the system to have dialogue memory capabilities, laying the technical foundation for subsequent multi-turn interactions.

[0170] Step S406: Obtain the stored historical entity labels and bounding box coordinates from the dialogue state machine to obtain historical data;

[0171] Step S407: Combine historical data with text questions in the current request message into context-enhanced input text;

[0172] Specifically, after extracting historical entity labels and coordinates from the dialogue state machine, the system performs a spatiotemporal semantic fusion operation to convert historical data into natural language descriptions, and then concatenates them with the current user question ([User] Where is the current focus?) using a separator to form enhanced input text.

[0173] Understandably, the essence of this step is to construct a prompting system that aligns with the cognitive framework of the visual language model: historical coordinate data is assigned semantic roles (such as location references) to guide the model in establishing a spatial relationship reasoning chain. For example, when a user asks for "focus location," the model automatically focuses on the corresponding area of ​​the image for fine-grained analysis based on the coordinate data of the historical tag "VIP area."

[0174] Step S408: Input the input text and the decoded image data in the current request message together into the fine-tuned multimodal large language model to generate new entity labels and corresponding bounding box coordinates, and obtain new data;

[0175] Specifically, the enhanced text and decoded image data are input into the model together to activate the multimodal joint inference engine: the visual encoder (such as ViT) parses the image into feature vectors, preserving pixel-level spatial information; the text encoder converts the enhanced cues into word vectors, where historical coordinate data is mapped to spatially perceptual vectors through a positional encoding layer; the cross-modal interaction module aligns image and text features through a cross-attention mechanism, enabling the model to automatically associate historical coordinate context when predicting new entities. For example, when historical data contains the location of "VIP Zone," the model will prioritize detecting the "focus icon" in its neighboring area, ultimately outputting the new entity label and bounding box coordinates.

[0176] Understandably, the key technical aspect of this step lies in the coordinate regression constraints injected during the LoRA fine-tuning stage. The model not only predicts the semantic meaning of the labels, but also outputs normalized coordinate values ​​(such as [0.25, 0.3, 0.65, 0.4]) through a fully connected layer, which are then converted into pixel coordinates ((640, 360, 1040, 480)) through inverse normalization, with the error rate controlled within 5%.

[0177] Step S409: After merging the new data with the historical data, encapsulate the data according to the preset structured data format, generate the final response message, and return it to the user client.

[0178] The structured encapsulation stage performs three layers of logical processing: First, data deduplication is checked by calculating the cosine similarity of word vectors between the new tag and the historical tag. If the similarity is greater than 0.8 and the intersection-over-union (IoU) of the bounding boxes is greater than 0.7, the entity is identified as a duplicate and filtered out. Then, incremental data merging is performed by appending the effective new data to the historical dataset. For example, if the original state machine contains ["watching TV"], adding ["games"] will update it to ["watching TV", "games"]. Finally, a response message is generated and the complete result is encapsulated in a preset XML or JSON format.

[0179] Step S410: Update the dialogue state machine bound to the session identifier according to the new data and store it in the in-memory database.

[0180] Specifically, the merged data is written into the dialogue state machine, and the session identifier is bound to the updated state machine and stored through the SETEX command of the in-memory database. The lifecycle is set, and persistent storage ensures that the next request can inherit the current full state of the dialogue, forming a technical closed loop of continuous interaction.

[0181] The above implementation constructs a complete technology chain for a conversation-aware visual question-answering system. By combining conversation identifiers with dialogue state machines, it breaks through the interaction limitations of traditional single-turn question-answering, enabling the model to have context tracking capabilities. Furthermore, the design of structured input and output protocols enables lossless conversion from raw data to business-parseable formats, while the dynamic context enhancement mechanism significantly improves target localization accuracy in complex scenarios through spatiotemporal semantic fusion.

[0182] Reference Figure 5 After the step of determining whether the session identifier is empty, the method further includes: if the session identifier is empty, then jump to step S501;

[0183] Step S501: Process the current request message as the first request and generate a unique session identifier;

[0184] The system calls a distributed unique identifier generator (such as one based on the snowflake algorithm or UUIDv4) to generate a globally unique session identifier.

[0185] In this embodiment, the process can employ a three-segment encoding structure of timestamp + node ID + sequence number, for example, generating sid-3f9c4e82. Its core logic lies in resolving the identifier conflict problem under multi-node concurrency. By pre-allocating node ID fields (e.g., allocating an independent ID segment to each machine in a server cluster) and utilizing a high-precision clock (nanosecond level) and a sequence number increment mechanism, it ensures that IDs generated by different nodes within the same millisecond are not duplicated. For example, when user A and user B simultaneously initiate their first request, the two servers generate sid-3f9c4e82 and sid-5d2a1b76 respectively, avoiding key-value overwriting issues during data storage.

[0186] Step S502: Create a new dialogue state machine on the server and initialize it to an empty set;

[0187] The server-side architecture is based on Finite State Machine (FSM) theory, creating a state storage structure with the current session ID as the key and initializing it as an empty set. For example, when a user uploads an image of the smart TV interface for the first time, the state machine will only record the tag and coordinates of the current request, without including irrelevant entities such as "game" or "VIP" from the historical conversation.

[0188] Step S503: Take the text question in the current request message as the input text, decode the current request message to obtain the current image data;

[0189] The system performs binary stream parsing on the current request message, separating the text query and image data. For example, it directly reads the text_query field from the message (such as "What are the categories in the navigation bar?") as plain text input; then it converts the Base64 encoded image data into an RGB pixel matrix, performs normalization operations, and performs block processing.

[0190] In addition, joint feature vectors can be generated by combining text encoders and visual encoders, for example, spatially associating the text “navigation bar” with the top horizontal region in an image.

[0191] Step S504: Input the input text and the current image data together into the fine-tuned multimodal large language model to generate an intermediate output result containing the newly added entity labels and their corresponding bounding box coordinates;

[0192] Specifically, after receiving input data, the model outputs entity categories (such as "watching TV", "VIP zone", "cloud karaoke") through a Softmax layer; at the same time, it uses a bounding box regressor to predict the target location, outputs normalized coordinate values ​​(such as x1=0.2, y1=0.3, x2=0.5, y2=0.7), and then restores them to absolute pixel coordinates based on the original image size.

[0193] It should be noted that the model can optimize the correlation between labels and coordinates through comparative learning of the loss function. For example, when the model mistakenly positions the coordinates of "VIP Zone" in the lower right corner of the image, the loss function will calculate the gradient based on the correct coordinates (central region) in the labeled data and backpropagate, forcing the model to correct spatial perception bias.

[0194] Step S505: Encapsulate the intermediate output results into a structured response message and return it to the user client, and carry a unique session identifier in the response message;

[0195] This involves adding a unique session identifier to the HTTP response header to establish session credentials for subsequent client requests. This step allows mobile or web clients to associate historical states simply by including this ID in the request header during subsequent interactions, without needing to repeatedly transmit the complete context data.

[0196] Step S506: Update the dialogue state machine based on the intermediate output results, and bind the updated dialogue state machine with a unique session identifier and store it in the memory database.

[0197] In the above implementation, from ensuring the global uniqueness of the session identifier and the pure initialization of the state machine, to the accurate alignment and joint reasoning of multimodal data, the engineering encapsulation and state persistence of the structured output are finally completed.

[0198] For example, in a smart TV navigation scenario, when a user first asks "What content is in the navigation bar?", the system not only accurately returns the labels and coordinates, but also establishes an initial spatiotemporal reference system for subsequent interactions. When the user further inquires "Where is the focus area?", the system can quickly locate the current focus area based on the stored coordinate data, providing a standardized paradigm for the engineering implementation of large models in dynamic interaction scenarios.

[0199] Reference Figure 6 As one implementation of step S407, the step of combining historical data with text questions in the current request message into context-enhanced input text includes:

[0200] Step S601: Extract the historical entity labels and corresponding bounding box coordinates recorded during the three most recent interactions from the historical data;

[0201] One key design element is the extraction of historical entity labels and corresponding bounding box coordinates from the dialogue state machine. This design is based on a technical mapping of the human short-term memory cognitive model. Cognitive science research indicates that the capacity of human working memory is typically 4±1 information chunks. The system accurately extracts three interaction records (such as the three most recently identified interface elements and their locations), avoiding distraction caused by excessive historical data while ensuring the integrity of key spatiotemporal cues.

[0202] Specifically, the system retrieves state machine storage units in reverse timestamp order, dynamically filtering timed-out data (e.g., retaining only records within 120 seconds) to ensure that the extracted entity labels (e.g., "VIP Zone," "Games," "Live Stream") and their bounding box coordinates (e.g., (664, 174, 1390, 282)) remain strongly correlated with the current user intent. This time-series extraction mechanism optimizes the model's computational efficiency and significantly reduces inference latency compared to the full history loading scheme.

[0203] Step S602: Concatenate the historical entity labels, bounding box coordinates, and recording time into historical context prompt text;

[0204] The historical context prompt text includes entity location information described in coordinate values;

[0205] Specifically, the extracted entity labels, bounding box coordinates, and recording time are concatenated into historical contextual prompt text, which is essentially to construct a spatiotemporal semantic field that can be understood by machines.

[0206] For example, this process can employ a three-layer transformation logic: First, a coordinate-semantic mapping engine transforms numerical coordinates into directional descriptions (e.g., mapping coordinates (68,184,132,275) to "left side of top navigation"), resolving the spatial perception barrier caused by purely numerical input; second, relative time identifiers (e.g., "[10 seconds ago]") are injected to explicitly establish event sequence relationships; finally, a natural language narrative is generated according to a preset template (e.g., "[12 seconds ago] The VIP area is located in the central area (664,174,1390,282); [8 seconds ago] The game icon is in the lower right corner (1200,650,1400,720)"). This structured transformation enables the model to associate discrete coordinate points with a continuous spatial scene, establishing a geographic reference system for subsequent reasoning.

[0207] Step S603: Concatenate the historical context prompt text with the text question in the current request message to form a continuous text sequence;

[0208] The key to this process is to concatenate the generated historical contextual prompts with the current user question (such as "Where is the current focus?") into a continuous text sequence, thereby achieving the coupling of spatiotemporal cues with immediate intent.

[0209] For example, a dual-delimiter architecture can be adopted, with [Context] marking historical data blocks (containing semantic descriptions of coordinates) and [Question] defining the boundaries of the current question (such as "User: Where is the focus?"). The two are physically separated but logically connected by line breaks.

[0210] Understandably, this design prevents semantic crosstalk (such as historical coordinate values ​​interfering with question keywords) while maintaining spatial relevance through implicit positional encoding. The system applies sinusoidal positional encoding to coordinate values ​​at the text input layer, enabling the model to autonomously establish a numerical spatial mapping relationship of "664 → x-axis center". When the user queries the focus location, the model automatically uses the historical coordinates of the "central region" as a spatial reference point for local scanning.

[0211] Step S604: Add a system role identifier at the beginning of the continuous text sequence to generate enhanced input text that conforms to the multi-turn dialogue input format.

[0212] Among them, a system role identifier (such as) is added at the beginning of the sequence. Essentially, it activates the instruction-following paradigm of the pre-trained model. The visual language model has already learned the system / user / assistant role division and identifiers during the pre-training phase. As a meta-instruction trigger, it forces the model to enter a multi-turn dialogue parsing mode.

[0213] Specifically, a global attention mechanism is established to give historical coordinate descriptions higher feature weights (such as improving the cross-attention score of the "central region"); at the same time, the internal state tracking module is activated to drive the model to perform context-based reasoning. For example, when an identifier exists, the model automatically associates historical coordinates with the current question type (such as focus query → spatial positioning task), and prioritizes searching for focus icons in the vicinity of the VIP zone coordinates, which greatly improves positioning accuracy.

[0214] In the above implementation, the model can establish spatial evolution logic based on historical coordinate data and accurately locate the target, greatly improving the response accuracy of complex interactive scenarios and providing an engineerable contextual solution for multimodal dialogue systems.

[0215] This application also discloses a multimodal large language model fine-tuning system.

[0216] A multimodal large language model fine-tuning system, the fine-tuning system comprising:

[0217] The preprocessing module is used to acquire the original business scene image and perform preprocessing to obtain the preprocessed image;

[0218] The structured annotation module is used to annotate bounding box coordinates and define semantic labels for entity targets in preprocessed images using annotation tools, and output structured annotation files.

[0219] The training sample set construction module is used to construct a training sample set including multi-turn text-to-image dialogue based on preprocessed images and structured annotation files;

[0220] The model loading module is used to load pre-trained multimodal large language models;

[0221] The fine-tuning instruction generation module is used to configure low-rank matrix decomposition parameters and generate a fine-tuning instruction set.

[0222] The model training module is used to input the training sample set into the pre-trained multimodal large language model, perform joint training operations based on the fine-tuning instruction set, and output the fine-tuned multimodal large language model.

[0223] The multimodal large language model fine-tuning system of this application embodiment can implement any of the above-mentioned fine-tuning methods, and the specific working process of each module in the fine-tuning system can refer to the corresponding process in the above-mentioned method embodiments.

[0224] In the several embodiments provided in this application, it should be understood that the provided methods and systems can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for example, the division of a certain module is merely a logical functional division, and in actual implementation there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0225] This application also discloses a computer device.

[0226] Computer equipment includes memory, processor, and computer program stored in memory and executable on the processor, wherein the processor executes the computer program to implement a multimodal large language model fine-tuning method as described above.

[0227] This application also discloses a computer-readable storage medium.

[0228] A computer-readable storage medium storing a computer program that can be loaded by a processor and executed as described above in any of the methods for fine-tuning a multimodal large language model.

[0229] The computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device; the program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0230] It should be noted that the computer device and storage medium in the embodiments of this application are respectively electronic devices and storage media that apply the above-described multimodal large language model fine-tuning method. Therefore, all embodiments of the above-described fine-tuning method are applicable to the computer device and storage medium, and can achieve the same or similar beneficial effects. For the computer device / storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple; relevant details can be found in the descriptions of the method embodiments.

[0231] In this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0232] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, disclosure, and appended claims in carrying out the claimed invention. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0233] The above are all preferred embodiments of this application and are not intended to limit the scope of protection of this application. Any feature disclosed in this specification (including the abstract and drawings) may be replaced by other equivalent or similar features unless specifically stated otherwise. That is, unless specifically stated otherwise, each feature is only one example of a series of equivalent or similar features.

Claims

1. A multi-modal large language model fine-tuning method, characterized in that, The fine-tuning method comprises: obtaining an original service scene image and preprocessing to obtain a preprocessed image; performing boundary box coordinate labeling and semantic label definition on the entity targets in the preprocessed image through a labeling tool, and outputting a structured labeling file; based on the preprocessed image and the structured labeling file, a training sample set comprising a multi-round image-text dialogue is constructed; loading a pre-trained multi-modal large language model, configuring low-rank matrix decomposition parameters, and generating a fine-tuning instruction set; inputting the training sample set into the pre-trained multi-modal large language model, performing joint training operation based on the fine-tuning instruction set, and outputting the multi-modal large language model after fine-tuning; the step of constructing a training sample set comprising a multi-round image-text dialogue based on the preprocessed image and the structured labeling file comprises: analyzing the semantic labels and corresponding boundary box coordinate data in the structured labeling file to generate a mapping relationship dictionary of entity semantic label sets and boundary box coordinate values; generating an initial question text and an initial answer text containing a structured identifier according to the entity semantic label sets in the mapping relationship dictionary; based on at least one entity semantic label in the initial answer text, generating a subsequent question text referring to the entity semantic label; retrieving the boundary box coordinate values of the referenced entity semantic labels according to the mapping relationship dictionary to generate a subsequent answer text containing corresponding coordinate identifiers; combining the initial question text, the initial answer text, the subsequent question text and the subsequent answer text into a continuous dialogue sequence; binding the continuous dialogue sequence, the storage path of the preprocessed image and the mapping relationship dictionary into a single training sample; aggregate all training samples and encode them into JSON format files by rows to generate a training sample set of multi-round image-text dialogue.

2. The multi-modal large language model fine-tuning method of claim 1, wherein, the step of inputting the training sample set into the pre-trained multi-modal large language model and performing joint training operation based on the fine-tuning instruction set comprises: performing convolutional coding on the preprocessed image in the training sample set to generate a visual feature vector; performing word embedding coding on the multi-round image-text dialogue text in the training sample set to generate a text feature vector; inputting the visual feature vector and the text feature vector into the cross-attention layer to output a fusion feature tensor; based on the boundary box coordinates in the structured labeling text, performing spatial coordinate regression on the fusion feature tensor to calculate a boundary box loss value; based on the semantic labels in the structured labeling file, performing entity classification on the fusion feature tensor to calculate a classification loss value; superimposing the boundary box loss value and the classification loss value to generate a joint loss function; according to the freeze instruction in the fine-tuning instruction set, locking all weight parameters of the multi-modal large language model; updating the weight increment based on the low-rank matrix decomposition parameters in the fine-tuning instruction set; optimizing the low-rank matrix decomposition parameters through back propagation until the joint loss function meets the preset convergence condition or reaches the preset training round, and outputting the multi-modal large language model after fine-tuning.

3. The multi-modal large language model fine-tuning method of claim 2, wherein, The spatial coordinate regression is performed on the fusion feature tensor, a boundary box loss value is calculated, and a Smooth L1 loss function is used; the entity classification is performed on the fusion feature tensor, a classification loss value is calculated, and a cross-entropy loss function is used.

4. The multi-modal large language model fine-tuning method according to any one of claims 1 to 3, characterized in that, The step of outputting the fine-tuned multi-modal large language model further comprises: The fine-tuned multi-modal large language model is packaged as a callable service interface; The input protocol of the service interface is defined as receiving a request message containing image data and a text question, and the output protocol is defined as returning structured response data containing at least one entity label and corresponding bounding box coordinates; A request message sent by a user client is received, and a session identifier in the request message is parsed; It is judged whether the session identifier is empty or not; if not, a dialog state machine bound to the session identifier is retrieved from a memory database on the server; The stored historical entity labels and bounding box coordinates are obtained from the dialog state machine to obtain historical data; The historical data and the text question in the current request message are combined into context-enhanced input text; The input text and the decoded image data in the current request message are jointly input into the fine-tuned multi-modal large language model to generate new entity labels and corresponding bounding box coordinates to obtain new data; The new data and the historical data are merged and packaged according to a preset structured data format to generate a final response message and return it to the user client; The dialog state machine bound to the session identifier is updated according to the new data and stored in the memory database.

5. The multi-modal large language model fine-tuning method of claim 4, wherein, The step of combining the historical data and the text question in the current request message into context-enhanced input text comprises: The historical entity labels and corresponding bounding box coordinates recorded in the last three interaction processes are extracted from the historical data; The historical entity labels, bounding box coordinates and record time are spliced into historical context prompt text; The historical context prompt text and the text question in the current request message are spliced into a continuous text sequence; A system role identifier is added at the beginning of the continuous text sequence to generate an enhanced input text in a multi-round dialogue input format.

6. The multi-modal large language model fine-tuning method of claim 4, wherein, After the step of judging whether the session identifier is empty or not, the following steps are further included: The current request message is processed as the first request to generate a unique session identifier; A new dialog state machine is created on the server and initialized as an empty set; The text question in the current request message is taken as input text, and the current image data is obtained by decoding the current request message; The input text and the current image data are jointly input into the fine-tuned multi-modal large language model to generate an intermediate output result containing new entity labels and corresponding bounding box coordinates; The intermediate output result is packaged as a structured response message and returned to the user client, and the unique session identifier is carried in the response message; The dialog state machine is updated according to the intermediate output result, and the updated dialog state machine is bound to the unique session identifier and stored in the memory database.

7. A multi-modal large language model fine-tuning system, characterized in that, The fine-tuning system comprises: A preprocessing module is configured to obtain an original business scenario image and perform preprocessing to obtain a preprocessed image; a structured annotation module configured to perform bounding box coordinate annotation and semantic label definition on the entity target in the preprocessed image through an annotation tool, and output a structured annotation file; a training sample set construction module configured to construct a training sample set including multi-turn image-text dialog based on the preprocessed image and the structured annotation file; a model loading module configured to load a pre-trained multi-modal large language model; a fine-tuning instruction generation module configured to configure low-rank matrix decomposition parameters and generate a fine-tuning instruction set; a model training module configured to input the training sample set into the pre-trained multi-modal large language model, perform joint training operation based on the fine-tuning instruction set, and output a multi-modal large language model after fine-tuning; the training sample set construction module is specifically configured to: analyze semantic labels and corresponding bounding box coordinate data in the structured annotation file, and generate a mapping relationship dictionary of entity semantic label set and bounding box coordinate value; generate an initial question text and an initial answer text containing a structured identifier according to the entity semantic label set in the mapping relationship dictionary; generate a subsequent question text referring to at least one entity semantic label in the initial answer text; retrieve the bounding box coordinate value of the referenced entity semantic label according to the mapping relationship dictionary, and generate a subsequent answer text containing a corresponding coordinate identifier; combine the initial question text, the initial answer text, the subsequent question text and the subsequent answer text into a continuous dialog sequence; bind the continuous dialog sequence, the storage path of the preprocessed image and the mapping relationship dictionary into a single training sample; aggregate all training samples and encode them into a JSON format file by rows to generate a training sample set of multi-turn image-text dialog.

8. A computer device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any one of claims 1 to 6 when executing the program.

9. A computer-readable storage medium, characterized in that: a computer program stored in the memory and executable on the processor, wherein the processor loads and executes the computer program to implement the method of any one of claims 1 to 6.

Images