High-efficiency and high-adaptability visual language model instruction fine tuning method

Abstract

The invention discloses a high-efficiency and high-adaptability visual language model instruction fine tuning method, which comprises the following steps of: extracting entity and visual feature attributes based on a multi-level analysis strategy, and calling a multi-strategy fusion algorithm to construct a structured entity prompt; jointly inputting the structured entity prompt, the image description text and the problem text into a large language model which is finely adjusted by an instruction to generate reasoning information; performing visual consistency verification operation on the reasoning information and calculating a confidence coefficient weight; integrating the training data and constructing a multi-task loss function, and based on the integrated training data and the multi-task loss function, performing fine tuning on the pre-training visual language model by adopting a hierarchical parameter updating strategy; processing a new image text input based on the fine-tuned model; according to the method, noise can be reduced based on structured entity extraction, cross-modal features are aligned based on layered fine-tuning balance capability and multi-task loss, and finally, comprehensive breakthrough in precision, efficiency and practicability is realized.

Description

Technical Field

[0001] This invention relates to the field of computer technology, specifically to the field of visual language models, and particularly to a highly efficient and adaptable method for fine-tuning visual language model instructions. Background Technology

[0002] In recent years, Visual Language Models (VLMs) have significantly improved accuracy in tasks such as visual question answering, while Large Language Models (LLMs) have demonstrated strong performance in reasoning tasks due to their ability to learn thought chains and context. Theoretically, VLMs should have reasoning capabilities similar to LLMs, but their actual performance is still limited by existing tuning methods. The mainstream approach is the "pre-training-fine-tuning" paradigm, which involves pre-training on a large dataset and then fine-tuning for downstream tasks. However, current VLM tuning techniques mostly rely on improving the network structure or simply adding instructions to existing datasets, which leads to problems such as complex implementation and insufficient model accuracy after fine-tuning, making it difficult to meet the needs of practical applications.

[0003] A Chinese patent (CN117975475A) proposes a method for fine-tuning instructions in a visual language model. This method generates image description text from the original dataset, uses an LLM (Visual Language Model) to generate inference information by combining questions and prompts, constructs a hybrid dataset, and fine-tunes the VLM using preset templates. This improves the model accuracy to some extent, but the method still has significant limitations, as follows: (1) Relying on LLM to generate inference information can easily introduce “illusionary” information that does not match the image content, affecting the accuracy of fine-tuning.

[0004] (2) The fine-tuning process only updates the parameters of the lightweight query converter, while the parameters of the image encoder and the pre-trained language model are frozen, resulting in insufficient adaptability of the model in some tasks.

[0005] (3) High requirements for hardware resources (although a single A100 GPU can complete the task, deployment is difficult in resource-constrained environments).

[0006] Therefore, there is an urgent need for a more efficient and adaptable method and system for fine-tuning VLM instructions to further improve model accuracy and generalization ability. Summary of the Invention

[0007] The purpose of this invention is to provide a highly efficient and adaptable method for fine-tuning visual language model instructions, thereby solving all or one of the aforementioned problems in the prior art.

[0008] To solve the above-mentioned technical problems, the specific technical solution of the present invention is as follows: This invention provides a highly efficient and adaptable method for fine-tuning visual language model instructions, comprising the following steps: Entity suggestion construction steps: Acquire raw image and text pair data, preprocess the raw image and text pair data, extract entity and visual feature attributes from the preprocessed data based on a multi-level analysis strategy, and call a multi-strategy fusion algorithm to construct structured entity prompts based on the entity and visual feature attributes. Inference information generation steps: Based on the image data processed by the entity prompt construction step and the pre-trained visual language model, image description text is generated; the user's question text is obtained, and the structured entity prompt, the image description text and the question text are input into a large language model that has been fine-tuned by instructions to generate inference information; Multi-level verification steps: The generated inference information is subjected to a visual consistency verification operation based on multidimensional verification to obtain retained inference information, and the confidence weight of the retained inference information is calculated. Training data integration steps: The original image and text pair data, the structured entity hints, the retained inference information, and the confidence weights are combined to obtain enhanced training data. Steps for constructing a multi-task loss function: Construct a multi-task loss function based on image-text contrast loss, entity alignment loss, and inference information consistency loss; Model fine-tuning steps: Based on the enhanced training data and the multi-task loss function, a hierarchical parameter update strategy is used to fine-tune the pre-trained visual language model. Steps for calling the model after fine-tuning: The model is fine-tuned to process new image and text inputs, extract entities and construct new structured prompts. Then, the new image and text and the new structured prompts are input into the fine-tuned model for cross-modal feature fusion, and inference is performed based on a progressive inference mechanism.

[0009] Furthermore, the multi-level analysis strategy includes: Define a named entity recognition module, and call the named entity recognition module to use a fine-tuned NER model to recognize object nouns in the text pair data as entities; Define an attribute extraction module, and call the attribute extraction module to determine the visual feature attributes associated with entities through dependency parsing and semantic role labeling.

[0010] Furthermore, the multi-strategy fusion algorithm includes: Basic combination strategy: Set the syntax structure and generate basic suggestions according to the syntax structure; Context enhancement strategy: Supplement the basic prompts with information based on the scene context; Visual saliency optimization strategy: The matching degree between the information processed by the context enhancement strategy and the original image is evaluated by a pre-trained visual language alignment model, and feature combination and filtering are performed based on the matching degree; Diversity control strategy: A hybrid approach of rule-based and learning-based methods is used to generate entity prompts.

[0011] Furthermore, the generation stage of the reasoning information further includes: The logical consistency between the generated content and the input information is evaluated using a natural language reasoning model, and the feature similarity between the generated text and the original image is calculated using a pre-trained visual language alignment model. Repeated or contradictory reasoning information is eliminated using a text deduplication algorithm.

[0012] Furthermore, the visual consistency verification operation based on multidimensional verification includes: defining a core verification module, and calling the core verification module to perform similarity verification based on visual language embedding space, fine-grained entity matching verification, and logical rationality verification respectively through three parallel processing channels; The similarity verification based on visual language embedding space includes: calculating the cosine similarity between text embeddings and image region feature embeddings in the inference information using a pre-trained CLIP model; The fine-grained entity matching verification includes: comparing the entities in the reasoning information with the structured entity hints item by item; The logical rationality verification includes: analyzing the logical coherence of the reasoning chain in the reasoning information through a natural language reasoning model.

[0013] Furthermore, the step of analyzing the logical coherence of the reasoning chain in the reasoning information using a natural language reasoning model further includes: Semantic analysis is performed on the reasoning information to identify the visual claims contained therein; For visual claims, a further verification process is performed, including: locating relevant objects in the corresponding image using an attention-based region feature extractor, verifying the object attributes of the relevant objects using an attribute classifier, verifying the quantity description using a counting network, verifying the orientation description using a spatial relationship network, and quantifying the output of the verification module into a confidence score. The confidence scores of the verification modules corresponding to the various visual claims are weighted and averaged to obtain the visual consistency score corresponding to each visual claim. A scoring threshold is set, and the reasoning information whose visual consistency score is higher than the scoring threshold is retained.

[0014] Furthermore, the multi-level verification step also includes: calculating the confidence weights of the retained inference information using an adaptive weighting algorithm; The adaptive weighting algorithm includes: linearly mapping the basic weights through the visual consistency scores; considering the type differences of inference information when calculating the weights; and reducing the weights of the later parts of a long inference chain based on a time decay factor.

[0015] Furthermore, the process of combining the original image and text pair data, the structured entity hints, the retained inference information, and the confidence weights to obtain enhanced training data includes: A unified data integration framework is established. Based on the unified data integration framework, data alignment processing, sample augmentation processing, and feature fusion processing are sequentially performed on the original image and text pair data, the structured entity prompts, the retained inference information, and the confidence weights to obtain structured training samples. The structured training samples are used as the augmented training data.

[0016] Furthermore, the image-text contrast loss is constructed based on an improved InfoNCE loss function; The entity alignment loss includes region entity comparison loss and attribute prediction loss; The reasoning information consistency loss includes logical coherence loss and visual basis loss; In the multi-task loss function construction step, the image-text comparison loss, the entity alignment loss, and the inference information consistency loss are combined using dynamic weighting coefficients to obtain the corresponding multi-task loss function.

[0017] Furthermore, the step of fine-tuning the pre-trained visual language model using a hierarchical parameter update strategy includes: Unfreeze the cross-attention layer parameters of the lightweight query converter in the pre-trained visual language model; keep the backbone parameters of the image encoder and text encoder in the pre-trained visual language model frozen; update the weights of the pre-trained visual language model through progressive parameter fusion.

[0018] The beneficial effects of the technical solution of this invention are: The high-efficiency and highly adaptable visual language model instruction fine-tuning method described in this invention can eliminate the "illusion" problem when large language models generate inference information through structured entity cue extraction and multi-level visual consistency verification, ensuring that the information is strictly based on the actual content of the image. It employs a hierarchical parameter update strategy (freezing the encoder backbone to retain general capabilities, unfreezing the cross-attention layer to learn task interactions, and progressive parameter fusion to balance pre-training and fine-tuning), balancing model generality and task specificity. Gradient caching and selective parameter updates reduce training memory usage, improving universality across different hardware environments. Based on structured entity extraction to reduce noise, hierarchical fine-tuning to balance capabilities, and multi-task loss alignment with cross-modal features, it ultimately achieves a comprehensive breakthrough in accuracy, efficiency, and practicality. Attached Figure Description

[0019] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the highly efficient and adaptable visual language model instruction fine-tuning method described in Embodiment 1 of the present invention; Figure 2 This is a detailed flowchart illustrating the highly efficient and adaptable visual language model instruction fine-tuning method described in Embodiment 1 of the present invention. Detailed Implementation

[0021] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0022] In the description of this invention, it should be noted that the embodiments described in this invention are only some embodiments of this invention, not all embodiments; based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0023] The terms "first," "second," etc., used in this specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0024] Example 1: This example provides a highly efficient and adaptable method for fine-tuning visual language model instructions, such as... Figure 1 and Figure 2 As shown, it includes the following steps: S100, Entity Hint Construction Steps: In this step, image-text pairs are first acquired, and the acquired data is preprocessed. Then, based on a multi-level analysis strategy, entities and their corresponding visual attributes are extracted from the preprocessed data. Based on these entities and visual attributes, a structured entity hint is constructed according to a multi-strategy fusion algorithm.

[0025] Optionally, the image-text pair data is obtained from real-world application scenarios or standard multimodal datasets, including but not limited to: COCO, Flickr30k, or Visual Genome; the above datasets contain high-quality images and their corresponding natural language descriptions (the description text contains rich visual entity information).

[0026] Optionally, in order to improve data quality, the acquired data is preprocessed, including but not limited to: removing low-quality samples and standardizing text format; since data preprocessing is not the focus of this application, the specific execution process will not be described in detail.

[0027] Optionally, in the entity extraction stage, a pre-trained language model based on the Transformer architecture (such as BERT or RoBERTa) is used to perform multi-level analysis on the preprocessed data, as follows: (i) Define a named entity recognition module and call the named entity recognition module to use a fine-tuned NER model to recognize specific object nouns (such as "apple", "car", etc.) in the text. These nouns usually correspond to salient visual objects in the image. (ii) Define an attribute extraction module, call the attribute extraction module to perform dependency parsing and semantic role labeling, and determine the visual feature attributes (such as color attributes, quantity attributes, spatial attributes, and state attributes) associated with the entity; among them, color attributes are adjectives (such as red, blue, etc.); quantity attributes are quantifiers (such as two, a group, etc.); spatial attributes are locative words (left, above, etc.); and state attributes are descriptive words (such as open, folded, etc.). Optionally, during the structured suggestion construction phase, a multi-strategy fusion algorithm (consisting of a basic combination strategy, a context enhancement strategy, a visual saliency optimization strategy, and a diversity control strategy) is executed, as follows: (i) Basic combination strategy: Set the syntax structure to "quantity + color + entity" and generate basic hints according to this syntax structure (such as "three brown puppies"); (ii) Context enhancement strategy: Combine scene context to supplement necessary information (e.g., in a kitchen scene, enhance "apple" to "red apple on the kitchen counter"); (iii) Visual saliency optimization strategy: Use a pre-trained visual-language alignment model (including but not limited to CLIP) to evaluate the matching degree between the generated information and the image, and select the most discriminative feature combination; (iv) Diversity control strategy: A hybrid approach based on rules and learning is used to generate the final entity cues, so that the generated cues cover the main visual features while maintaining appropriate expressive diversity, providing more effective supervision signals for subsequent model fine-tuning.

[0028] S200, Inference Information Generation Steps: In this step, image description text is generated based on the image data processed by S100 and the pre-trained visual language model. Then, based on the structured entity prompts, the currently generated image description, and the question text provided by the user, it is input into a large language model that has been fine-tuned by instructions to generate inference information.

[0029] Optionally, in the image description generation stage, a visual language model (including but not limited to BLIP-2 or Flamingo) pre-trained on large-scale multimodal data is used. The image processed by S100 is input into the visual language model, and hierarchical image features are extracted through its visual encoder (usually ViT or CLIP-ViT). Then, it interacts with the language decoder through a cross-modal attention mechanism to generate coherent image description text.

[0030] Optionally, in the image description generation stage, to improve the description quality, a beam search algorithm is used for text decoding, and a temperature parameter is set to control the generation diversity, ultimately ensuring that the description accurately covers the main content of the image and matches the needs of subsequent inference generation.

[0031] Optionally, in the reasoning information generation stage, structured entity hints, image descriptions, and question text are input into a large language model (including but not limited to GPT-4 or LLaMA-2) that has been fine-tuned by instructions. The specific processing flow is as follows: (i) Set a specific prompt template, based on which the input information is organized into structured input (e.g., "Based on the image description '{description text}' and entity information '{entity prompt}', answer the question '{question text}' and provide a step-by-step reasoning process"); (ii) Employ chain-of-thought stimulating technology to guide the model to generate a complete logical chain containing intermediate reasoning steps; (iii) By using a nucleus sampling strategy, the creativity and reliability of the generated results are balanced.

[0032] Optionally, in the reasoning information generation stage, to ensure the quality of the generated reasoning information, a multi-level verification mechanism (including semantic consistency check, visual basis verification, and redundancy filtering) is adopted, as follows: (i) Semantic consistency check: The logical consistency between the generated content and the input information is evaluated using a natural language reasoning model (such as DeBERTa); (ii) Visual basis verification: The feature similarity between the generated text and the original image is calculated using a pre-trained visual-language alignment model; (iii) Redundancy filtering: The text deduplication algorithm eliminates duplicate or contradictory reasoning information, and the final retained reasoning information together with the structured entity prompts of S100 constitutes an enhanced training signal.

[0033] S300, Multi-level Validation Steps: In this step, the inference information obtained in S200 is subjected to visual consistency verification (including similarity verification based on visual-language embedding space, fine-grained entity matching verification, and logical rationality verification), thereby retaining the inference information that matches the image content and calculating its confidence weight.

[0034] Specifically, in the preparation phase, a multi-level verification framework is established, receiving inference information from S200 and raw image-text pair data from S100.

[0035] Optionally, in visual consistency verification, a core verification module is defined, which contains three parallel processing channels, as follows: (1) Similarity verification based on visual-language embedding space: The cosine similarity between the inference information text embedding and the image region feature embedding is calculated by using the pre-trained CLIP model; (2) Fine-grained entity matching verification: The entities mentioned in the reasoning information are compared with the structured entity hints of S100 item by item; (3) Logical rationality verification: The logical coherence of the reasoning chain in the reasoning information is analyzed through natural language reasoning models (including but not limited to DeBERTa); in specific implementation, the reasoning information is first semantically parsed to identify the visual claims contained therein, such as "there are three brown dogs in the image"; then, for each visual claim, a verification module is defined and the verification module is called to execute the following verification process: (3.1) Locate relevant objects in the corresponding image using a region feature extractor (including but not limited to DETR) based on an attention mechanism; (3.2) Verify the object attributes of the relevant object through an attribute classifier (e.g., confirm "brown" using a color classifier); (3.3) Verify the quantity description using a counting network (e.g., confirm “three”); (3.4) Verify the location description (e.g., "on a bench") through a spatial relationship network; (3.5) Finally, the output of each verification module is quantified into a confidence score of 0-1, and the visual consistency score of the claim is obtained by weighted averaging.

[0036] Optionally, in the confidence weight calculation stage of the inference information, an adaptive weighting algorithm is used, as follows: (i) Linearly map the underlying weights through visual consistency scores; (ii) Consider the differences in the types of inference information (e.g., assign a higher weight to factual descriptions than to speculative descriptions); (iii) Introduce a time decay factor to appropriately reduce the weight of the later part of a longer inference chain.

[0037] Optionally, after the confidence weight calculation stage, the inference information with a visual consistency score higher than the threshold (usually set to 0.7) is retained, and the confidence weight corresponding to the inference information is recorded for use in subsequent steps.

[0038] Specifically, the reasoning information verified in this step, together with the original image-text pairs and structured entity cues, constitutes a reliable training sample, providing high-quality supervision signals for subsequent model fine-tuning.

[0039] S400, Training Data Integration Steps: In this step, the original image-text pair data, structured entity cues, verified inference information, and their confidence weights from the previous steps are combined to form augmented training data.

[0040] Optionally, the original image-text pairs are used as the base training samples; structured entity cues, such as “brown wooden table”; and validated inference information and its confidence weights, such as “because the table is wooden, it is suitable for placing heavy objects”, with a weight of 0.85.

[0041] Optionally, a unified data integration framework is established. Based on this framework, the multimodal data from the aforementioned steps undergoes a three-stage processing (data alignment stage, sample augmentation stage, and feature fusion stage), ultimately organizing it into structured training samples, as follows: (i) Data alignment stage: The spatiotemporal attention mechanism ensures the positional correspondence of image regions, text descriptions, entity cues and reasoning information in the feature space; the attention weights between different modal elements are calculated through the transformer-based cross-modal alignment module, thereby establishing an accurate mapping relationship; (ii) Sample augmentation stage: Dynamically sample the inference information based on confidence weights, retain high-weight inference information (weighted confidence > 0.8); generate variants of medium-weight inference information (weighted confidence 0.6-0.8) through semantically preserved data augmentation algorithms (such as synonym replacement and sentence transformation); filter or downweight low-weight inference information (weighted confidence < 0.6). (iii) Feature fusion stage: Encode different data elements into a unified representation; wherein, grid features in the image are extracted by a pre-trained visual encoder (including but not limited to ViT); word-level embeddings of text descriptions and entity prompts are obtained by a language model (including but not limited to BERT); and sequence dependencies of inference information are captured by a bidirectional LSTM.

[0042] Optionally, to ensure the balance and diversity of the training data during the three-stage processing, class-balanced sampling, hard example mining, and noise filtering are performed, as follows: (i) Category-balanced sampling: Based on the image content category, the sampling probability of different samples is adjusted; (ii) Difficult example mining: Automatically identify samples whose model predictions are inconsistent and increase the proportion of such samples; (iii) Noise filtering: Eliminate contradictory data combinations through consistency checks.

[0043] Optionally, the final augmented training data generated in this step is stored in a standardized format, specifically including the following fields: (1) Image feature vector; (2) The original text tag sequence; (3) Entity prompt embedding; (4) Inference information sequence and its weight labels; (5) Cross-modal attention mask, etc.

[0044] Specifically, the enhanced training data generated in this step not only retains the authenticity of the original data, but also significantly improves the quality of the supervision signal through structured prompts and verified inference information, laying a solid foundation for subsequent model fine-tuning.

[0045] S500, Multi-task Loss Function Construction Steps: In this step, a multi-task loss function is constructed based on image-text contrast loss, entity alignment loss, and inference information consistency loss.

[0046] Optional, image-text contrast loss (L cont This is implemented using an improved InfoNCE loss function; specifically, in the calculation, this loss maps image features v and text features t to the joint embedding space through a projection head, and calculates the normalized temperature-scaled similarity, as follows: ; Among them, L cont This is used to establish local region-phrase correspondences using the aforementioned structured entity cues while calculating global image-text pair similarity, and to dynamically weight negative samples using the aforementioned confidence weights to reduce the impact of potentially erroneous negative samples on training; specifically, L cont s(v,t) is the image-text contrast loss; v is the image feature vector; t is the text feature vector (positive sample); s(v,t) is the similarity function; τ is the learnable temperature parameter; t' is the negative sample text feature; v′ is the negative sample image feature; The similarity between the text features of all negative samples and image v is weighted and summed. The similarity between the features of all negative sample images and the text tt is weighted and summed.

[0047] Optional, entity alignment loss (L ent The loss consists of a region-entity contrast loss (requiring each entity cue to be as close as possible to the image region it describes in the feature space) and an attribute prediction loss (predicting attributes such as color and quantity of entities using an auxiliary classification head). Specifically, this loss is calculated using a triplet loss with edges, as follows: ;; Among them, L entThis is specifically used to optimize the correspondence between the aforementioned structured entity cues and image regions; specifically, L ent p is the entity alignment loss; v is the entity cue feature vector; + Features of positive sample image regions; v − d(p, v) represents the negative sample image region features; d(p, v) is the distance function used to calculate the distance between features p and v (such as Euclidean distance or cosine distance); α is the boundary value (margin), a preset constant used to control the minimum interval between positive and negative samples.

[0048] Optional, inference information consistency loss (L reas The design for the aforementioned reasoning information includes a logical coherence loss (used to evaluate the logical reasonableness of the generated reasoning through a pre-trained language model) and a visual basis loss (used to ensure that each claim in the reasoning can find corresponding visual evidence); specifically, this loss is implemented in the form of weighted cross-entropy, as follows: ;; Among them, w i The confidence weights calculated in step S103, y i For the real labels, f(x) i ) represents the model's predicted probability.

[0049] Optionally, this step ultimately combines the three loss functions using dynamic weighting coefficients to obtain the corresponding multi-task loss function (L). total ),as follows: ;; The weight coefficients λ are automatically adjusted using an uncertain weighting method to balance the learning progress of different tasks; during backpropagation, gradient pruning and hierarchical learning rate strategies are employed to ensure training stability; this L total It not only makes full use of the high-quality supervision signals generated in the preceding steps, but also ensures the synergistic optimization of each learning objective through scientific loss combination, which can steadily improve the performance of the final model.

[0050] S600, Model Fine-tuning Steps: In this step, based on the enhanced training data and multi-task loss function constructed in the previous steps, a hierarchical parameter update strategy is adopted to fine-tune the pre-trained visual language model, including: unfreezing the cross-attention layer parameters of the lightweight query converter, keeping the backbone parameters of the image encoder and text encoder frozen, and updating the model weights through progressive parameter fusion.

[0051] Optionally, in the fine-tuning phase, the model architecture is first decomposed into modules, dividing the complete visual language model into three parameter levels, as follows: (1) Basic coding layer: This layer is the backbone network containing image encoder and text encoder. The parameters of this layer are kept frozen, thus preserving the general representation ability obtained by pre-training; (2) Interaction Adaptation Layer: This layer is mainly the cross-attention module in the lightweight query converter. The parameters of this layer are completely unfrozen, and then the task-specific cross-modal interaction mode is learned. For this unfrozen cross-attention layer, a phased training strategy is adopted, that is: in the initial stage, only the parameters of the query and key projection matrices are updated, and the value projection matrix and attention output matrix are gradually unfrozen in the later stage. This layered unfreezing scheme can effectively prevent overfitting while ensuring that the model gradually adapts to the target task.

[0052] (3) Fusion output layer: This layer includes a classification head and a regression head. This layer is fine-tuned using a progressive update strategy.

[0053] Optionally, in this step, progressive parameter fusion is the core innovation, and its logic specifically includes: calculating the current fine-tuning parameter θ after every k training iterations (usually k=1000). t The weighted fusion with the pre-training initial parameters θ0 is as follows: ; The mixing coefficient α(t) is scheduled using cosine annealing. T is the total number of training steps; These are the parameters after fusion; These are the initial parameters for pre-training; These are the fine-tuned parameters at the current training step number t; The mixing coefficients change with time t, controlling the fusion ratio of new and old parameters. Progressive parameter fusion can not only retain the generalization ability of the pre-trained model, but also absorb the task-specific knowledge obtained from fine-tuning.

[0054] It should be noted that this step uses a gradient caching mechanism to cache and reuse the forward propagation results of the frozen layer. This not only optimizes training efficiency but also reduces computational overhead, maintains model stability, and improves its performance in downstream tasks such as visual question answering and image / text retrieval.

[0055] S700, steps for calling the model after fine-tuning: In this step, new image-text inputs are processed based on the fine-tuned model, entities are extracted to construct structured cues, and then the new image-text and structured cues are input into the fine-tuned visual language model for cross-modal feature fusion. Finally, inference is performed based on a progressive inference mechanism.

[0056] Optionally, the structured entity extraction stage operates in the same manner as step S100, as follows: (i) Analyze the input text using a pre-trained named entity recognition module to identify key object nouns (including but not limited to bicycle, red balloon, etc.). (ii) Extract visual feature descriptors (such as color, quantity, and spatial relationships) through an attribute parsing network; (iii) Generate structured entity hints according to the standardized syntax structure ([quantity][color][object name][location]).

[0057] Optionally, the cross-modal feature fusion stage is executed through three parallel processing pathways (visual pathway, text pathway, and interaction pathway), as follows: (1) Visual pathway: Multi-scale visual features are extracted by freezing the image encoder, with particular attention paid to image regions related to structured cues; (2) Text path: The original text and structured prompts are jointly encoded through a bidirectional attention mechanism; (3) Interactive pathway: Fine-grained modal associations are established through the unfrozen cross-attention layer.

[0058] Among them, the three-path architecture mentioned above not only retains the powerful representational capabilities of the pre-trained model, but also leverages the cross-modal interaction advantages obtained through fine-tuning.

[0059] Optionally, in the reasoning phase, a progressive reasoning mechanism is adopted, as follows: (i) First round of reasoning: generating preliminary hypotheses based on global image-text matching; (ii) During verification, structured cues are used to locate key visual evidence for verification; (iii) Correction phase: The reasoning results are gradually corrected through iterative optimization.

[0060] Optionally, the above process invokes a confidence calibration strategy. When an intermediate result is detected to have low confidence (i.e., confidence < 0.6), the reasoning information generation mechanism in step S200 is triggered to assist in decision-making.

[0061] Optionally, based on the fine-tuned model, the final output can be a task-related main answer (such as the answer to VQA), key visual evidence supporting the answer, and an interpretable reasoning chain, thereby improving the reliability and interpretability of the model output.

[0062] As a preferred embodiment of the present invention, when the present invention is applied to the field of automotive fault diagnosis, the following implementation process is involved: (i) First, the image of the faulty part of the car and the text description of the repair technician are received as input data; when an image-text description of “engine shaking at idle speed, accompanied by fault code P0302” is input, this method performs named entity recognition based on the BERT model finely tuned in the automotive field and extracts information such as “spark plug”, “oxygen sensor” and “fault code P0302”.

[0063] (ii) Subsequently, the fault characteristics are further analyzed through dependency analysis, including: extracting the status attributes from "shaking at idle speed", associating the specific meaning of "cylinder 2 misfire" from the fault code; and combining this information into a standardized structured prompt, such as "[engine][idle speed][shaking][fault code][P0302][cylinder 2 misfire]".

[0064] (iii) Next, in the image description generation stage, a visual language model pre-trained specifically for the automotive field is used to analyze the input engine compartment image and generate a detailed text description, such as "carbon deposits exist in the ignition coil of the second cylinder in the engine compartment, and the spark plug electrode gap is too large". This process is combined with visual saliency detection technology to ensure that the generated description accurately corresponds to the key areas in the image.

[0065] (iv) Then, the structured prompts, image descriptions and repair questions are input into a large language model enhanced with automotive repair knowledge to generate a step-by-step fault reasoning chain. The model may output the following: "1. Fault code P0302 indicates misfire in cylinder 2; 2. Image shows ignition coil contamination and abnormal spark plug gap; 3. Both of these factors together cause incomplete combustion, resulting in vibration."

[0066] (v) To ensure the reliability of the generated content, multi-level visual consistency verification is performed: First, the specific components in the image (such as the ignition coil area of ​​cylinder 2) are located using an object detection model, and then the visual similarity score between the text description and the image area is calculated; at the same time, a fine-grained classification model is used to verify the existence of specific attributes such as "carbon deposition", and physical parameters such as spark plug gap are measured using image segmentation technology; these verification results are quantified into confidence scores between 0 and 1, and after weighted calculation, the final credibility assessment is formed (for example, a certain inference chain may obtain a comprehensive confidence score of 0.88).

[0067] (vi) During the reasoning phase, when a new fault case is input, the system first generates structured prompts to locate key visual evidence, and then gradually improves the diagnostic conclusion through a progressive reasoning mechanism. Finally, it outputs the cause of the fault, maintenance suggestions, and visual evidence and logical reasoning chain that support the conclusion (e.g., for the case of ABS warning light, the system can not only point out the possibility of "wheel speed sensor failure", but also highlight the suspicious wiring harness damage area in the image and provide a step-by-step diagnostic reasoning process).

[0068] It should be noted that during the model training phase, the system constructs a loss function that includes multiple task objectives. Among them, the contrastive loss focuses on optimizing the matching degree between local regions of the image and related text descriptions, the entity alignment loss ensures that the parts mentioned in the text correspond accurately to the actual objects in the image, and the reasoning consistency loss uses confidence as the weight to supervise the model to learn reasonable fault reasoning logic. These loss functions will dynamically adjust the weight ratio according to the task difficulty during the training process to achieve a more balanced optimization. In addition, the model uses a hierarchical parameter update strategy for fine-tuning, keeping the backbone network parameters of the image and text encoders fixed, and only unfreezing some parameters of the cross-modal attention layer; this design not only retains the general representation ability of the pre-trained model, but also effectively learns the cross-modal interaction patterns unique to the automotive fault diagnosis task.

[0069] In summary, when applied to the field of automotive fault diagnosis, this method can significantly improve the accuracy and efficiency of automotive fault diagnosis. Compared with traditional end-to-end training methods, the hierarchical fine-tuning strategy of this method can not only reduce the training resource requirements, but also ensure excellent inference performance.

[0070] It should be noted that all the examples above are merely for explaining the present invention and should not be construed as limiting the scope of protection of the present invention.

[0071] Example 2: This example, based on the same inventive concept as the high-efficiency and highly adaptable visual language model instruction fine-tuning method described in Example 1, provides a high-efficiency and highly adaptable visual language model instruction fine-tuning system, including: The entity prompting construction module is used to: acquire raw image and text pair data, preprocess the raw image and text pair data, extract entity and visual feature attributes from the preprocessed data based on a multi-level analysis strategy, and call a multi-strategy fusion algorithm to construct structured entity prompts based on the entity and visual feature attributes. The reasoning information generation module is used to: generate image description text based on the image data processed by the entity prompting construction module and the pre-trained visual language model; obtain the user's question text, and input the structured entity prompts, the image description text and the question text into a large language model that has been fine-tuned by instructions to generate reasoning information; A multi-level verification module is used to: perform a visual consistency verification operation based on multi-dimensional verification on the generated inference information to obtain retained inference information, and calculate the confidence weight of the retained inference information; The training data integration module is used to: combine the original image and text pair data, the structured entity cue, the retained inference information, and the confidence weights to obtain enhanced training data; The multi-task loss function construction module is used to: construct multi-task loss functions based on image-text contrast loss, entity alignment loss, and inference information consistency loss; The model fine-tuning module is used to: fine-tune the pre-trained visual language model based on the enhanced training data and the multi-task loss function, using a hierarchical parameter update strategy. The fine-tuned model invocation module is used to: process new image and text inputs based on the fine-tuned model, extract entities and construct new structured prompts, then input the new image and text and the new structured prompts into the fine-tuned model for cross-modal feature fusion, and perform inference based on a progressive inference mechanism.

[0072] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0073] Example 3: This example provides a computer-readable storage medium, including: The storage medium is used to store computer software instructions used to implement the high-efficiency and highly adaptable visual language model instruction fine-tuning method described in Embodiment 1 above. It includes a program for executing the program set up for the high-efficiency and highly adaptable visual language model instruction fine-tuning method. Specifically, the executable program can be built into the high-efficiency and highly adaptable visual language model instruction fine-tuning system described in Embodiment 2. In this way, the high-efficiency and highly adaptable visual language model instruction fine-tuning system can implement the high-efficiency and highly adaptable visual language model instruction fine-tuning method described in Embodiment 1 by executing the built-in executable program.

[0074] Furthermore, the computer-readable storage medium in this embodiment can be any combination of one or more readable storage media, wherein the readable storage medium includes an electrical, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof.

[0075] Unlike existing technologies, this application employs a highly efficient and adaptable visual language model instruction fine-tuning method. This method eliminates the "illusion" problem in large language models generating inference information through structured entity cues extraction and multi-level visual consistency verification, ensuring that information is strictly based on the actual content of the image. It utilizes a hierarchical parameter update strategy (freezing the encoder backbone to retain general capabilities, unfreezing the cross-attention layers to learn task interactions, and progressive parameter fusion to balance pre-training and fine-tuning), balancing model generality and task specificity. Gradient caching and selective parameter updates reduce training memory usage, improving universality across different hardware environments. Based on structured entity extraction to reduce noise, hierarchical fine-tuning to balance capabilities, and multi-task loss alignment with cross-modal features, this method achieves a comprehensive breakthrough in accuracy, efficiency, and practicality.

[0076] It should be understood that in the various embodiments of this document, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.

[0077] It should also be understood that, in the embodiments herein, the term "and / or" is merely a description of the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following associated objects have an "or" relationship.

[0078] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this document.

[0079] In the embodiments provided herein, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, devices, or units, or they may be electrical, mechanical, or other forms of connection.

[0080] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments described herein, depending on actual needs.

[0081] Furthermore, the functional units in the various embodiments of this document can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0082] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this paper, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this paper. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0083] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A highly efficient and adaptable method for fine-tuning visual language model instructions, characterized in that, Includes the following steps: Entity prompting construction steps: acquire original image and text pair data, preprocess the original image and text pair data, extract entity and visual feature attributes from the preprocessed data based on a multi-level analysis strategy, and call a multi-strategy fusion algorithm to construct structured entity prompts based on the entity and visual feature attributes. Inference information generation step: Based on the image data processed in the entity prompting construction step and the pre-trained visual language model, generate image description text; The user's question text is obtained, and the structured entity hints, the image description text, and the question text are input together into a large language model that has been fine-tuned by instructions to generate inference information. Multi-level verification steps: Perform visual consistency verification based on multi-dimensional verification on the generated inference information to obtain retained inference information, and calculate the confidence weight of the retained inference information; Training data integration step: Combine the original image and text pair data, the structured entity cue, the retained inference information, and the confidence weights to obtain enhanced training data; Steps for constructing a multi-task loss function: Construct a multi-task loss function based on image-text contrast loss, entity alignment loss, and inference information consistency loss; Model fine-tuning steps: Based on the enhanced training data and the multi-task loss function, fine-tune the pre-trained visual language model using a hierarchical parameter update strategy; The fine-tuned model invocation steps are as follows: Based on the fine-tuned model, new image and text inputs are processed, entities are extracted and new structured prompts are constructed. Then, the new image and text and the new structured prompts are input into the fine-tuned model for cross-modal feature fusion, and inference is performed based on a progressive inference mechanism.

2. The high-efficiency and highly adaptable visual language model instruction fine-tuning method according to claim 1, characterized in that: The multi-level analysis strategy includes: Define a named entity recognition module, and call the named entity recognition module to use a fine-tuned NER model to recognize object nouns in the text pair data as entities; Define an attribute extraction module, and call the attribute extraction module to determine the visual feature attributes associated with entities through dependency parsing and semantic role labeling.

3. The high-efficiency and highly adaptable visual language model instruction fine-tuning method according to claim 1, characterized in that: The multi-strategy fusion algorithm includes: Basic combination strategy: Set the syntax structure and generate basic suggestions according to the syntax structure; Context enhancement strategy: Supplement the basic prompts with information based on the scene context; Visual saliency optimization strategy: The matching degree between the information processed by the context enhancement strategy and the original image is evaluated by a pre-trained visual language alignment model, and feature combination and filtering are performed based on the matching degree; Diversity control strategy: A hybrid approach based on rules and learning is used to generate entity prompts.

4. The high-efficiency and highly adaptable visual language model instruction fine-tuning method according to claim 1, characterized in that: The generation stage of the reasoning information further includes: The logical consistency between the generated content and the input information is evaluated using a natural language reasoning model, and the feature similarity between the generated text and the original image is calculated using a pre-trained visual language alignment model. Repeated or contradictory reasoning information is eliminated using a text deduplication algorithm.

5. The high-efficiency and highly adaptable visual language model instruction fine-tuning method according to claim 1, characterized in that: The visual consistency verification operation based on multidimensional verification includes: defining a core verification module, and calling the core verification module to perform similarity verification based on visual language embedding space, fine-grained entity matching verification, and logical rationality verification respectively through three parallel processing channels; The similarity verification based on visual language embedding space includes: calculating the cosine similarity between text embeddings and image region feature embeddings in the inference information using a pre-trained CLIP model; The fine-grained entity matching verification includes: comparing the entities in the reasoning information with the structured entity prompts item by item; The logical rationality verification includes: analyzing the logical coherence of the reasoning chain in the reasoning information through a natural language reasoning model.

6. The high-efficiency and highly adaptable visual language model instruction fine-tuning method according to claim 5, characterized in that: The step of analyzing the logical coherence of the reasoning chain in the reasoning information using a natural language reasoning model further includes: Semantic analysis is performed on the reasoning information to identify the visual claims contained therein; For visual claims, a further verification process is performed, including: locating relevant objects in the corresponding image using an attention-based region feature extractor, verifying the object attributes of the relevant objects using an attribute classifier, verifying the quantity description using a counting network, verifying the orientation description using a spatial relationship network, and quantifying the output of the verification module into a confidence score. The confidence scores of the verification modules corresponding to the various visual claims are weighted and averaged to obtain the visual consistency score corresponding to each visual claim. A scoring threshold is set, and the reasoning information whose visual consistency score is higher than the scoring threshold is retained.

7. The high-efficiency and highly adaptable visual language model instruction fine-tuning method according to claim 6, characterized in that: The multi-level verification step further includes: using an adaptive weighted algorithm to calculate the confidence weight of the retained inference information; The adaptive weighting algorithm includes: linearly mapping the basic weights through the visual consistency scores; considering the type differences of inference information when calculating the weights; and reducing the weights of the later parts of a long inference chain based on a time decay factor.

8. The high-efficiency and highly adaptable visual language model instruction fine-tuning method according to claim 1, characterized in that: The process of combining the original image and text pair data, the structured entity hints, the retained inference information, and the confidence weights to obtain enhanced training data includes: A unified data integration framework is established. Based on the unified data integration framework, data alignment processing, sample augmentation processing, and feature fusion processing are sequentially performed on the original image and text pair data, the structured entity prompts, the retained inference information, and the confidence weights to obtain structured training samples. The structured training samples are used as the augmented training data.

9. The high-efficiency and highly adaptable visual language model instruction fine-tuning method according to claim 1, characterized in that: The image-text contrast loss is constructed based on the improved InfoNCE loss function; The entity alignment loss includes region entity comparison loss and attribute prediction loss; The reasoning information consistency loss includes logical coherence loss and visual basis loss; In the multi-task loss function construction step, the image-text comparison loss, the entity alignment loss, and the inference information consistency loss are combined using dynamic weighting coefficients to obtain the corresponding multi-task loss function.

10. The high-efficiency and highly adaptable visual language model instruction fine-tuning method according to claim 1, characterized in that: The method of fine-tuning the pre-trained visual language model using a hierarchical parameter update strategy includes: Unfreeze the cross-attention layer parameters of the lightweight query converter in the pre-trained visual language model; keep the backbone parameters of the image encoder and text encoder in the pre-trained visual language model frozen; update the weights of the pre-trained visual language model through progressive parameter fusion.

Images