Interaction method based on multi-modal large language model, electronic device and storage medium

Abstract

The application provides an interactive method based on a multi-modal large language model, an electronic device and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring image data and question data associated with the image data; performing visual interaction task processing on the image data and the question data based on a preset multi-modal large language model to obtain a multi-modal interaction result of the question data; and the multi-modal large language model is obtained by alignment training based on visual dependency and evidence consistency driven training sample data. The visual dependency is used to represent the degree of dependence of the visual interaction task on the visual evidence, and the evidence consistency is used to represent the alignment degree between the multi-modal interaction result and the visual evidence. The application can improve the accuracy and reliability of the multi-modal large language model in performing the visual interaction task.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an interaction method, electronic device and storage medium based on a multimodal large language model. Background Technology

[0002] Among related technologies, the Multimodal Large Language Model (MLLM) can achieve basic visual language interaction functions, but it still has significant technical defects in the deep understanding of visual information. It is difficult to interact based on a true understanding of the visual content. Especially in open-domain question answering tasks, it is easy for the model to give answers based on language priors. That is, the model relies on the prior knowledge inherent in the language modality to respond, rather than giving interaction results based on the real visual information input. This results in the model's response not matching the actual visual content and failing to truly reflect the objective information in the visual scene. Summary of the Invention

[0003] The main objective of this application is to propose an interaction method, electronic device, and storage medium based on a multimodal large language model, aiming to improve the accuracy and reliability of the multimodal large language model in performing visual interaction tasks.

[0004] To achieve the above objectives, a first aspect of this application proposes an interaction method based on a multimodal large language model, the method comprising: Acquire image data and query data associated with the image data; The image data and the question data are processed for visual interaction tasks based on a preset multimodal large language model to obtain the multimodal interaction results of the question data. The multimodal large language model is obtained by alignment training based on training sample data driven by visual dependence and evidence consistency. The visual dependence is used to characterize the degree of dependence of the visual interaction task on visual evidence, and the evidence consistency is used to characterize the degree of alignment between the multimodal interaction results and visual evidence.

[0005] In some embodiments, the method further includes: Acquire multimodal dialogue sample data; The multimodal dialogue sample data is subjected to visual dependency assessment to obtain a visual dependency score, and the multimodal dialogue sample data is subjected to evidence consistency assessment to obtain an evidence consistency score. Training sample data is constructed based on the visual dependence score and the evidence consistency score; The multimodal large language model is aligned and trained based on the training sample data to obtain the trained multimodal large language model.

[0006] In some embodiments, the multimodal dialogue sample data includes: image sample data and question sample data associated with the image sample data. The step of performing visual dependency assessment on the multimodal dialogue sample data to obtain a visual dependency score includes: Based on a preset multidimensional visual dependency quantification index, visual dependency assessment processing is performed on the image sample data and the question sample data to obtain a multidimensional visual dependency score vector; the visual dependency score includes the multidimensional visual dependency score vector.

[0007] In some embodiments, the visual dependency assessment processing of the image sample data and the question sample data based on a preset multidimensional visual dependency quantification index includes: Extract visual features from the image sample data, and convert the question sample data into semantic features; The visual features and the semantic features are fused to obtain joint visual-semantic features; The visual-semantic joint features are evaluated in parallel using a preset multi-task regression head to obtain a multi-dimensional visual dependency score vector output by the multi-task regression head; the multi-task regression head corresponds one-to-one with the multi-dimensional visual dependency quantification index.

[0008] In some embodiments, the multimodal dialogue sample data further includes: initial response candidate data of the question sample data, and the evidence consistency evaluation processing of the multimodal dialogue sample data to obtain an evidence consistency score includes: The intersection-union ratio (IU) of the visual evidence region and the visual attention region corresponding to the initial response candidate data is calculated to obtain the IU data between the visual evidence region and the visual attention region. A positional consistency evaluation process is performed on the text fragments in the initial response candidate data and the text box recognition results in the image sample data to obtain a positional consistency score between the text fragments and the text box recognition results. The intersection-union ratio data and the positional consistency score are normalized to obtain the evidence consistency score; the weight ratio of the positional consistency score is positively correlated with the number of text boxes represented by the text box recognition result.

[0009] In some embodiments, constructing training sample data based on the visual dependency score and the evidence consistency score includes: Based on the visual dependence score, the evidence consistency score, and the preset phased adjustable parameters, sampling weights are generated; Dynamic sampling processing is performed based on the sampling weights to obtain training sample data.

[0010] In some embodiments, the step of performing dynamic sampling processing based on the sampling weights to obtain training sample data includes: Obtain training sample batches by performing dynamic sampling based on the sampling weights; Based on the positive response training samples that meet the visual dependence requirements in the batch of training samples, near-sense negative response training samples are constructed. Preference sample pairs are generated based on the positive response training samples and the near-sense negative response training samples; the training sample data includes the preference sample pairs.

[0011] In some embodiments, constructing near-negative response training samples based on positive response training samples that meet the visual dependence requirement in the training sample batch includes at least one of the following: Text perturbation is added to the positive response training samples to obtain near-negative response training samples; Visual evidence is removed from the positive response training samples to obtain near-sense negative response training samples. By performing reverse reasoning on the positive response training samples, near-sense negative response training samples are obtained.

[0012] In some embodiments, the alignment training of the multimodal large language model based on the training sample data includes: Based on the sampling weights and the regularization constraints corresponding to the evidence consistency scores, an alignment preference loss is generated. The training sample data is input into the multimodal large language model for alignment training, and the model parameters of the multimodal large language model are optimized based on the alignment preference loss.

[0013] To achieve the above objectives, a second aspect of this application proposes an interactive system based on a multimodal large language model, the system comprising: The acquisition module is used to acquire image data and query data associated with the image data; The model interaction module is used to perform visual interaction task processing on the image data and the question data based on a preset multimodal large language model to obtain the multimodal interaction result of the question data; the multimodal large language model is obtained by alignment training based on training sample data driven by visual dependence and evidence consistency, the visual dependence is used to characterize the degree of dependence of the visual interaction task on visual evidence, and the evidence consistency is used to characterize the degree of alignment between the multimodal interaction result and the visual evidence.

[0014] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the interaction method based on a multimodal large language model described in the first aspect.

[0015] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the interaction method based on a multimodal large language model as described in the first aspect.

[0016] To achieve the above objectives, a fifth aspect of the present application provides a computer program product, which includes a computer program that, when executed by a processor, implements the interaction method based on a multimodal large language model provided in the first aspect above.

[0017] The interaction method, system, electronic device, computer-readable storage medium, and computer program product based on a multimodal large language model proposed in this application acquire image data and question data associated with the image data; perform visual interaction task processing on the image data and the question data based on a preset multimodal large language model to obtain the multimodal interaction result of the question data; the multimodal large language model is obtained by alignment training based on training sample data driven by visual dependence and evidence consistency, wherein the visual dependence is used to characterize the degree of dependence of the visual interaction task on visual evidence, and the evidence consistency is used to characterize the degree of alignment between the multimodal interaction result and the visual evidence.

[0018] This application embodiment uses training sample data driven by visual dependence and evidence consistency to perform alignment training on a multimodal large language model to obtain a trained multimodal large language model. Here, visual dependence is used to characterize the degree of dependence of the visual interaction task on visual evidence, and evidence consistency is used to characterize the degree of alignment between the multimodal interaction result and the visual evidence. Then, when performing the visual interaction task, the acquired image data and the question data associated with the image data are input into the trained multimodal large language model, so as to perform visual interaction task processing on the image data and the question data based on the multimodal large language model to obtain the multimodal interaction result of the question data.

[0019] Thus, this application embodiment performs alignment training on a multimodal large language model based on training sample data driven by visual dependence and evidence consistency. This systematically suppresses the language prior eavesdropping phenomenon of the multimodal large language model through a fine-grained control system on the data side, thereby improving the perception and reasoning ability of the multimodal large language model in high visual dependence tasks. As a result, when performing visual interaction tasks, the multimodal large language model provides interaction results based on the real visual information input, improving the accuracy and credibility of the multimodal large language model in performing visual interaction tasks. Attached Figure Description

[0020] Figure 1 A flowchart illustrating the steps of the interaction method based on a multimodal large language model provided in some embodiments of this application; Figure 2 A flowchart illustrating the steps of the interaction method based on a multimodal large language model provided in this application in other embodiments; Figure 3 The diagram shows the structure and workflow of the visual dependency scorer involved in some embodiments of the interaction method based on a multimodal large language model provided in the embodiments of this application. Figure 4 for Figure 2 A detailed flowchart of step S203; Figure 5 for Figure 2 A detailed flowchart of step S204; Figure 6 The interaction method based on a multimodal large language model provided in the embodiments of this application is illustrated in some embodiments with schematic diagrams of the algorithm flow. Figure 7 A schematic diagram of the technical architecture involved in a complete embodiment of the interaction method based on a multimodal large language model provided in this application embodiment; Figure 8 A schematic diagram illustrating the accuracy comparison of the interaction method based on a multimodal large language model provided in an embodiment of this application in a complete embodiment; Figure 9 A schematic diagram comparing sample efficiency in a complete embodiment of the interaction method based on a multimodal large language model provided in this application embodiment; Figure 10 A schematic diagram of ablation experiment results involved in a complete embodiment of the interaction method based on a multimodal large language model provided in this application embodiment; Figure 11 A schematic diagram of the structure of an interactive system based on a multimodal large language model provided in an embodiment of this application; Figure 12This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0022] It should be noted that although functional modules are divided in the device / system schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device / system or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0024] First, the relevant technical terms involved in the embodiments of this application will be explained.

[0025] Visual dependence: Visual dependence, also known as strong-visuality (SV), is used to characterize the degree to which a visual interaction task relies on visual evidence. The visual dependence score—SV score—is used to measure the scalar / vector metric of the sample's reliance on visual evidence. The SV score can integrate multiple dimensions such as Optical Character Recognition (OCR) text density, geometric / table complexity, occlusion, number of instances, and layout structure entropy to quantify the richness and complexity of the visual information required to answer the question.

[0026] Evidence Consistency (EC): Evidence consistency is used to characterize the degree of alignment between multimodal interaction results and visual evidence. EC score is an indicator used to measure the degree of alignment between multimodal interaction results and visual evidence regions or text boxes. EC score can quantify the empirical reliability of the model's response based on techniques such as page alignment, text box matching, or gradient-weighted class activation mapping (Grad-CAM).

[0027] Semantically-Near Negative (SNN) Samples: The semantics of the questions given by SNN and the positive response samples are highly similar, and the language answers of SNN are self-consistent but lack sufficient visual support or conflict with graphical evidence. The core of SNN is used to explicitly penalize the reliance of multimodal large language models on language shortcuts in preference learning.

[0028] Preference pairs: Preference sample pairs can be characterized as<r+ , r-> , where r+ is the correct response expected by the multimodal large language model, i.e., a positive response sample (consistent with visual evidence), and r- is a negative response not expected by the multimodal large language model (usually a near-sense negative response sample SNN). The preference sample pairs are used for contrastive alignment objectives such as Direct Preference Optimization (DPO) / Reinforcement Learning from Human Feedback (RLHF).

[0029] Hardness Catalysis Hard example catalysis is a process that intensifies the training distribution along a strong visual dimension by automatically constructing or retrieving SNNs. The goal of hard example catalysis is to focus the training of multimodal large language models on samples that require strong visual reasoning to answer correctly.

[0030] SV-Aware Sampler: The SV-aware sampler is an adaptive dynamic sampling strategy based on SV scores and the training phase. The SV-aware sampler, which uses a proportionally skewed sampling subset of data, ensures that key samples with high SV and low EC can be fully learned by the multimodal large language model.

[0031] Task-Bucket Weighting: An adaptive quota allocation mechanism is implemented for the combined application of optical character recognition (OCR) technology with chart data extraction, understanding, and structuring, document visual question answering (DocVQA), scene counting / relational reasoning, and other strong visual task clusters to ensure the minimum proportion and dynamic balance of strong visual tasks in the training dataset.

[0032] Next, the overall concept of the interaction method based on a multimodal large language model provided in the embodiments of this application will be briefly described.

[0033] Artificial intelligence technology is rapidly evolving towards multimodal fusion. Multimodal Language Modeling (MLLM), with its outstanding cross-modal understanding and generation capabilities, excels in tasks such as open-domain question answering and complex visual language understanding, and has become a core research and application direction in the field of artificial intelligence. By organically combining a visual encoder with a large language model, MLLM effectively breaks down the barriers between visual and linguistic information, enabling end-to-end multimodal interaction in diverse scenarios such as image understanding, multi-turn dialogue generation, and visual logic reasoning. This provides crucial technical support for functional upgrades in fields such as intelligent question answering, intelligent cockpits, and intelligent security.

[0034] However, while MLLM models can achieve basic visual-language interaction functions, they still have significant technical deficiencies in the deep understanding of visual information. They are difficult to interact based on a true understanding of the visual content. Especially in open-domain question answering tasks, they are prone to language prior knowledge-based responses, that is, the model relies on the prior knowledge inherent in the language modality to respond, rather than giving interaction results based on the real visual information input. This results in the model's response not matching the actual visual content and failing to truly reflect the objective information in the visual scene.

[0035] The linguistic prior-based "sneak answer" flaw commonly found in open-domain question-answering tasks in MLLM models means that when the evidence provided by the image is insufficient or the model fails to effectively utilize visual information, MLLM models can still generate seemingly reasonable but visually unfounded answers by leveraging their powerful language modeling capabilities and statistical prior knowledge of the corpus. This linguistic shortcut phenomenon persists in alignment techniques such as DPO, especially when the training data contains a large number of weak visual samples. In these cases, the gradient updates of MLLM models are often primarily driven by the language side, leading to insufficient optimization of visual representations. MLLM models struggle to form a true perception-reasoning-expression closed loop, thus limiting their verifiability and interpretability.

[0036] Meanwhile, in numerous high-value real-world applications, such as industrial form reading, financial statement understanding, medical image analysis, and warehouse counting and defect detection, higher demands are placed on the OCR capabilities, spatial layout awareness, and geometric and logical reasoning abilities of MLLM models. These tasks not only require MLLM models to possess accurate visual recognition capabilities but also demand that they be able to understand semantics and express them accurately in conjunction with context. However, current MLLM models still struggle to achieve the goal of both understanding and accurately expressing in these scenarios. Therefore, how to improve the visual representation learning quality of MLLM models and enhance the coupling effect between their visual reasoning and language generation is gradually becoming a key scientific and engineering problem driving the development of next-generation highly reliable multimodal intelligent systems.

[0037] To address the aforementioned issues, this application proposes an interaction method, system, electronic device, computer-readable storage medium, and computer program product based on a multimodal large language model. The aim is to resolve the problem of insufficient reliance on visual evidence in MLLM model alignment training, particularly the tendency to fall into language shortcuts when handling highly visual tasks, thereby improving the accuracy and reliability of multimodal large language models in performing visual interaction tasks.

[0038] In this embodiment, a multimodal large language model is trained using training sample data driven by visual dependence and evidence consistency to obtain a trained multimodal large language model. Here, visual dependence is used to characterize the degree of dependence of the visual interaction task on visual evidence, and evidence consistency is used to characterize the degree of alignment between the multimodal interaction result and the visual evidence. Then, when performing the visual interaction task, the acquired image data and the question data associated with the image data are input into the trained multimodal large language model, so as to perform visual interaction task processing on the image data and the question data based on the multimodal large language model to obtain the multimodal interaction result of the question data.

[0039] Thus, this application embodiment performs alignment training on a multimodal large language model based on training sample data driven by visual dependence and evidence consistency. This systematically suppresses the language prior eavesdropping phenomenon of the multimodal large language model through a fine-grained control system on the data side, thereby improving the perception and reasoning ability of the multimodal large language model in high visual dependence tasks. As a result, when performing visual interaction tasks, the multimodal large language model provides interaction results based on the real visual information input, improving the accuracy and credibility of the multimodal large language model in performing visual interaction tasks.

[0040] Furthermore, in this embodiment, by constructing a strong visual SV quantization scale (visual dependence) and combining it with evidence consistency EC evaluation, a universal sampler and hard example generator that can be used across data sources and tasks is realized. This allows the training process of multimodal large language models to accurately focus on those samples that have the highest requirements for visual reasoning.

[0041] Moreover, compared to scaling up the model or performing expensive re-pre-training, the training of multimodal large language models in this application is low-cost and highly integrated. It can be embedded as training middleware into existing human preference optimization DPO / supervised fine-tuning SFT pipelines. Through efficient training with small samples, significant improvements can be achieved for visually demanding tasks such as OCR & Chart, DocVQA, and counting and localization.

[0042] Furthermore, in the embodiments of this application, by using SV-Aware active sampling and SNN catalysis, the distribution of training data can be fundamentally changed, forcing the multimodal large language model to rely on visual evidence during training, thereby systematically suppressing language shortcuts and significantly improving the empirical reliability of the model.

[0043] Finally, the method provided in this application has immediate practical value for tasks such as reading and monitoring, tool use, and dial / tag recognition in embodied intelligence. It can improve the perception and decision-making capabilities of multimodal large language models in complex real-world scenarios, thus having broad application prospects.

[0044] In conjunction with the above, this application's embodiments introduce for the first time an active sampling and near-sense negative response sample catalysis mechanism driven by SV and EC metrics in the preference alignment training of the MLLM model. This can transform the empirical observation that more visually biased samples have greater benefits for DPO into a measurable, schedulable, and reusable data routing strategy and generation mechanism. Thus, through fine-grained control on the data side, the language shortcut phenomenon of the MLLM model can be systematically suppressed without changing the network structure, significantly improving the empirical reliability and performance of the MLLM model on strong visual tasks.

[0045] Next, the interaction method, system, electronic device, computer-readable storage medium, and computer program product based on a multimodal large language model provided in this application will be specifically described through the following embodiments, and firstly, the various detailed embodiments of the interaction method based on a multimodal large language model provided in this application will be described in detail.

[0046] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0047] It should be noted that the interaction method based on a multimodal large language model provided in this application relates to the field of artificial intelligence technology. The interaction method based on a multimodal large language model provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be an intelligent agent (such as an embodied robot), an in-vehicle terminal, or an electronic device such as a smartphone, tablet, laptop, or desktop computer that is associated with a vehicle and can communicate and interact with the vehicle via a network. The server can be the backend server terminal device of the terminal, which can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The software can be an application implementing the interaction method based on a multimodal large language model, a computer program, and a storage medium carrying the computer program. It should be understood that, based on different design needs of practical applications, the terminals, servers, and software that apply the interaction method based on the multimodal large language model provided in the embodiments of this application may also be other forms not listed here. The interaction method based on the multimodal large language model provided in the embodiments of this application does not specifically limit these.

[0048] Furthermore, this application can also be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: embodied intelligent agents, in-vehicle terminals, personal computers, server computers, handheld or portable devices, tablet devices, embodied intelligent systems (such as multiprocessor systems, microprocessor-based systems, etc.), set-top boxes, programmable consumer electronics devices, personal computers (PCs), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via communication networks. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0049] For ease of understanding and explanation, the following text will use the example of a terminal device performing corresponding physical operations on an intelligent agent in a complex 3D real physical environment, applying the interaction method based on a multimodal large language model provided in the embodiments of this application, to describe in detail the various specific embodiments of this application. In some descriptions, the terminal device may be simply referred to as a terminal. The implementation of the interaction method based on a multimodal large language model provided in the embodiments of this application for any of the above-described subject matter can refer to the implementation process of the interaction method based on a multimodal large language model described below.

[0050] Please refer to Figure 1 , Figure 1 The flowchart illustrates the steps of the interaction method based on a multimodal large language model provided in some embodiments of this application. It should be understood that, although... Figure 1 The flowcharts illustrating subsequent steps show the execution order of some method steps. However, based on different design needs in practical applications, the interaction method based on a multimodal large language model provided in this application embodiment can, of course, adopt an execution order of method steps different from that shown in the figures. That is, Figure 1 The order of the method steps shown does not constitute a limitation on the execution logic order of the interaction method based on a multimodal large language model provided in the embodiments of this application. Any other method based on... Figure 1 Reasonable changes to the order of steps shown should be included within the protection scope of the interaction method based on a multimodal large language model provided in the embodiments of this application.

[0051] like Figure 1 As shown, in some embodiments, the interaction method based on a multimodal large language model provided in this application may include steps S101 and S102 as shown below.

[0052] Step S101: Obtain image data and query data associated with the image data.

[0053] It should be noted that the question data associated with the image data can be a question automatically generated by the terminal device during the execution of an interactive task based on needs (such as behavioral reasoning) and related to the content displayed in the image data, or a question actively input by a human user into the terminal device related to the content displayed in the image data. For example, the question data could be "What color is the cup in the picture?", etc.

[0054] When a terminal device performs a visual interaction task, it first obtains the image data required to perform the task and the question data associated with the image data.

[0055] In some embodiments, the image data may be environmental image data obtained by the terminal device through real-time image acquisition of the physical environment by built-in / external sensors.

[0056] In other embodiments, the image data may also be image data actively uploaded by human users and received by the terminal device based on the human-computer interaction interface (HMI).

[0057] Step S102: Perform visual interaction task processing on the image data and the question data based on a preset multimodal large language model to obtain the multimodal interaction result of the question data; the multimodal large language model is obtained by alignment training based on training sample data driven by visual dependence and evidence consistency, the visual dependence is used to characterize the degree of dependence of the visual interaction task on visual evidence, and the evidence consistency is used to characterize the degree of alignment between the multimodal interaction result and the visual evidence.

[0058] Terminal devices may require training sample data driven by visual dependence and evidence consistency to perform alignment training on multimodal large language models. That is, firstly, training sample data is constructed by combining the degree of dependence of visual interaction tasks on visual evidence in the representation of visual dependence, and the degree of alignment between multimodal interaction results and visual evidence in the representation of evidence consistency. Then, the multimodal large language model is aligned and trained using this training sample data to obtain the trained multimodal large language model.

[0059] In this way, when performing visual interaction tasks, the terminal device can input the acquired image data and the question data associated with the image data into the trained multimodal large language model, and then perform visual interaction task processing on the image data and the question data based on the multimodal large language model to obtain the multimodal interaction result generated by the multimodal large language model for the question data.

[0060] In this embodiment, the terminal device first constructs training sample data by combining the degree of dependence of the visual interaction task on visual evidence (represented by visual dependency) and the degree of alignment between the multimodal interaction results and visual evidence (represented by evidence consistency). Then, this training sample data is used to train the multimodal large language model, resulting in a trained multimodal large language model. Subsequently, when performing a visual interaction task, the terminal device first acquires the image data required for the task and the associated question data. It then inputs the image data and question data into the trained multimodal large language model. Based on this model, the device processes the visual interaction task for the image data and question data, obtaining the multimodal interaction result generated by the model for the question data.

[0061] Thus, this application embodiment performs alignment training on a multimodal large language model based on training sample data driven by visual dependence and evidence consistency. This systematically suppresses the language prior eavesdropping phenomenon of the multimodal large language model through a fine-grained control system on the data side, thereby improving the perception and reasoning ability of the multimodal large language model in high visual dependence tasks. As a result, when performing visual interaction tasks, the multimodal large language model provides interaction results based on the real visual information input, improving the accuracy and credibility of the multimodal large language model in performing visual interaction tasks.

[0062] Please refer to Figure 2 , Figure 2 The flowcharts illustrating the steps of the interaction method based on a multimodal large language model provided in this application are shown in other embodiments.

[0063] like Figure 2 As shown, in some embodiments, the interaction method based on a multimodal large language model provided in this application may further include steps S201 to S204 as shown below.

[0064] Step S201: Obtain multimodal dialogue sample data.

[0065] When a terminal device performs alignment training on a multimodal large language model using training sample data driven by visual dependence and evidence consistency, it first acquires multimodal dialogue sample data.

[0066] In some embodiments, multimodal dialogue sample data may include image sample data and question sample data associated with the image sample data. For example, the image sample data is image I, and the question sample data is a question instruction q for image I.

[0067] In some embodiments, the multimodal dialogue sample data may include, in addition to image sample data and question sample data associated with the image sample data, initial response candidate data for the question data. For example, the initial response candidate data may include a positive response sample r+ to the question instruction q.

[0068] In some embodiments, the multimodal dialogue sample data may also include OCR results and layout structure.

[0069] Step S202: Perform visual dependency assessment processing on the multimodal dialogue sample data to obtain a visual dependency score, and perform evidence consistency assessment processing on the multimodal dialogue sample data to obtain an evidence consistency score.

[0070] After acquiring the multimodal dialogue sample data, the terminal device further performs visual dependency assessment on the multimodal dialogue sample data to obtain a visual dependency score for the multimodal dialogue sample data. At the same time, the terminal device also performs evidence consistency assessment on the multimodal dialogue sample data to obtain an evidence consistency score for the multimodal dialogue sample data.

[0071] In some embodiments, the terminal device can use a preset strong visual SV scorer to perform visual dependency evaluation processing on image sample data and question sample data associated with the image sample data in multimodal dialogue sample data, and obtain a visual dependency score for the multimodal dialogue sample data. For example, the terminal device can use the SV scorer to calculate the SV score vector on the input (I, q) to obtain the visual dependency score s(I, q)∈Rk.

[0072] In some embodiments, the terminal device can use a preset evidence consistency evaluator (EC) to perform evidence consistency evaluation on the initial response candidate data in the multimodal dialogue sample data, and obtain the evidence consistency score of the multimodal dialogue sample data. For example, the terminal device can use the EC evaluator to calculate the evidence consistency score e(I, q, r) ∈ [0, 1] for the initial response candidate data r.

[0073] Step S203: Construct training sample data based on the visual dependence score and the evidence consistency score.

[0074] After obtaining the visual dependency score and evidence consistency score of the multimodal dialogue sample data, the terminal device further performs dynamic sampling based on the visual dependency score and evidence consistency score to obtain training sample data.

[0075] In some embodiments, the terminal device may employ a preset adaptive dynamic sampling strategy. By combining visual dependency scores and evidence consistency scores of multimodal dialogue sample data, diverse and comprehensive training sample data are sampled in the early stage of training to promote multimodal generalization. In the later stage of training, difficult examples with high visual complexity (High SV) and low evidence consistency (Low EC) are sampled as training sample data to guide the multimodal large language model from language shortcuts to reasoning based on deep visual evidence.

[0076] Step S204: Alignment training is performed on the multimodal large language model based on the training sample data to obtain the trained multimodal large language model.

[0077] After the terminal device constructs the training sample data, it inputs the training sample data into the initial multimodal large language model for alignment training, and continuously optimizes the model parameters of the multimodal large language model during the training process until the multimodal large language model converges, thereby obtaining the trained multimodal large language model.

[0078] In this embodiment, multimodal dialogue sample data is acquired through a terminal device, and visual dependency assessment is performed on the multimodal dialogue sample data to obtain a visual dependency score. Simultaneously, the terminal device also performs evidence consistency assessment on the multimodal dialogue sample data to obtain an evidence consistency score. Then, dynamic sampling is performed based on the visual dependency score and the evidence consistency score to obtain training sample data. Finally, the training sample data is input into the initial multimodal large language model for alignment training, and the model parameters of the multimodal large language model are continuously optimized during the training process until the multimodal large language model converges, thereby obtaining the trained multimodal large language model.

[0079] Thus, this embodiment constructs a strong visual dependency score vector (SV score) for image-text question answering and multimodal dialogue samples, quantifies the sample's dependency on visual information, and combines it with the evidence consistency score (EC score) to form a joint metric SV and EC, which is used to evaluate the visual dependency and evidence consistency of the samples. This enables a universal sampler and hard example generator that can be used across data sources and tasks, allowing the training process of the multimodal large language model to accurately focus on the samples with the highest requirements for visual reasoning, thereby effectively improving the visual representation learning quality of the multimodal large language model and enhancing the coupling effect between its visual reasoning and language generation.

[0080] In some embodiments, where the multimodal dialogue sample data includes image sample data and question sample data associated with the image sample data, the step of "performing visual dependency assessment processing on the multimodal dialogue sample data to obtain a visual dependency score" in step S202 above may include the following steps: Based on a preset multidimensional visual dependency quantification index, visual dependency assessment processing is performed on the image sample data and the question sample data to obtain a multidimensional visual dependency score vector; the visual dependency score includes the multidimensional visual dependency score vector.

[0081] It should be noted that multidimensional visual dependency quantification metrics can include OCR density and quality, geometric and chart complexity, target occlusion and complexity, instance count and counting difficulty, and layout structure entropy. Specifically, OCR density and quality quantify the number of text boxes per unit area (density), number of characters, and mean and variance of OCR confidence scores. Geometric and chart complexity quantifies the number of charts (such as bar charts, pie charts, and line charts) and tables, and the detection scores of table headers / axis / grid lines. Target occlusion and complexity quantifies the occlusion / overlap ratio of key targets (such as OCR text boxes and instances), which can be estimated by a lightweight detection model. Instance count and counting difficulty quantifies the number of instances (such as objects, people, and animals) in an image, serving as a proxy indicator for the difficulty of multi-object scenes. Layout structure entropy quantifies the histogram entropy based on the spatial distribution of text boxes / primitives; the higher the entropy value, the more irregular or complex the layout structure.

[0082] When performing visual dependency assessment on multimodal dialogue sample data, the terminal device can quantify the multidimensional visual dependency metrics of both the image sample data and the associated question sample data within the multimodal dialogue sample data. This quantification yields a multidimensional visual dependency score vector. The terminal device can then use this multidimensional visual dependency score vector as the visual dependency score for the multimodal dialogue sample data.

[0083] For example, the multidimensional visual dependency rating vector can be a strongly visual SV rating vector. When performing visual dependency evaluation on multimodal dialogue sample data, the terminal device can define an SV rating vector s = [for image sample data I and the question sample data q associated with that image sample data I]. ]∈R5. Wherein, The OCR density and quality score of sample (I, q) is a quantitative score. A high score means that the image contains a large amount of text information that needs to be recognized and read, such as tables, documents, etc. The quantification of the geometric and graphical complexity of the sample (I, q) is significant because a high score indicates that the problem requires complex geometric reasoning or data extraction (such as trend analysis, data comparison, etc.). The quantification of target occlusion and complexity for sample (I, q) means that a high score indicates that visual information is interfered with or incomplete, requiring stronger robust reasoning. The number of instances of a sample (I, q) is quantified by the difficulty of counting. A high score is significant because the problem involves precise counting or complex relational reasoning. The quantification of the page structure entropy of sample (I, q) is significant because a high score is required to understand complex page structures (such as multi-column documents, infographics, etc.).

[0084] In some embodiments, the step of "performing visual dependency evaluation processing on the image sample data and the question sample data based on a preset multidimensional visual dependency quantification index" may include the following steps: Extract visual features from the image sample data, and convert the question sample data into semantic features; The visual features and the semantic features are fused to obtain joint visual-semantic features; The visual-semantic joint features are evaluated in parallel using a preset multi-task regression head to obtain a multi-dimensional visual dependency score vector output by the multi-task regression head; the multi-task regression head corresponds one-to-one with the multi-dimensional visual dependency quantification index.

[0085] It's important to note that the Multi-Task Regression Head (MTRH) is an extension of the Regression Head. Unlike the Single-Task Regression Head, which outputs only a single continuous value, the MTRH can jointly model different regression objectives through multiple parallel branches. It is a mainstream technical solution for solving the problem of predicting continuous values ​​for multiple objectives. In deep learning, the MTRH, as a key component of the Multi-Task Learning (MTL) architecture, specifically refers to the top-level output layer module of the model capable of simultaneously processing multiple regression tasks. Its core function is to predict multiple continuous numerical objectives in parallel, rather than discrete class probabilities, based on a general representation output by a shared feature extractor.

[0086] When a terminal device quantifies image sample data and question sample data associated with the image sample data in multimodal dialogue sample data based on multidimensional visual dependency quantification index for visual dependency assessment, it can first extract the visual features of the image sample data and convert the question sample data into semantic features. Then, it can perform multimodal fusion of the visual features and the semantic features to obtain the fused visual-semantic joint features. Finally, it can perform visual dependency assessment on the visual-semantic joint features in parallel through a multi-task regression head to obtain the multidimensional visual dependency score vector output by the multi-task regression head. Among these, the task regression head corresponding to the OCR density and quality metric quantifies the OCR density and quality of image sample data and question sample data separately; the task regression head corresponding to the geometric and graph complexity metric quantifies the geometric and graph complexity of image sample data and question sample data separately; the task regression head corresponding to the target occlusion and complexity metric quantifies the target occlusion and complexity of image sample data and question sample data separately; the task regression head corresponding to the instance number and counting difficulty metric quantifies the instance number and counting difficulty of image sample data and question sample data separately; and the task regression head corresponding to the layout structure entropy metric quantifies the layout structure entropy of image sample data and question sample data separately.

[0087] In some embodiments, the terminal device can use the SV Scorer, employing a lightweight visual encoder combined with a multi-task regression head, to quantify the multidimensional visual dependency quantification index of image sample data and question sample data respectively, thereby obtaining a multidimensional visual dependency score vector.

[0088] For example, such as Figure 3 As shown, the specific structure and workflow of the visual dependency scorer SV Scorer may include: Feature extraction layer: Input image sample data via lightweight visual encoder Extracting multi-scale visual representations and questioning sample data. via embedded encoder The vectors are converted into semantic feature vectors, which provide the basic representation for subsequent fusion. Freeze copies of the parameters of the visual encoder for a pre-trained multimodal large model to maintain consistency in the feature space.

[0089] Cross-Modality Fusion: Employs a lightweight attention mechanism to align and fuse visual and semantic features, generating joint visual-semantic features. This is used to comprehensively represent the correlation between visual structure and question semantics, and serve as the input for score regression.

[0090] Multi-task regression head: In visual-semantic joint features Based on this, parallel regression outputs a five-dimensional scoring vector: These reflect the OCR dependency, geometric and graphical complexity, occlusion complexity, number of instances and counting difficulty, and the intensity of page structure entropy of the sample (I, q), respectively.

[0091] In some embodiments, such as Figure 3 As shown, the SV scorer can also use a small number of manually labeled samples to optimize the regression head with L2 regression loss, and combine pseudo-labels to expand unlabeled data for bootstrapping training. The training objective can be defined as: .

[0092] Thus, once the multimodal large language model converges as a whole, the SV scorer can be used to automatically calculate SV scores for large-scale samples for sampling and measurement purposes.

[0093] In some embodiments, where the multimodal dialogue sample data further includes initial response candidate data of the question sample data, the step of "performing evidence consistency evaluation processing on the multimodal dialogue sample data to obtain an evidence consistency score" in step S202 above may include the following steps: The intersection-union ratio (IU) of the visual evidence region and the visual attention region corresponding to the initial response candidate data is calculated to obtain the IU data between the visual evidence region and the visual attention region. A positional consistency evaluation process is performed on the text fragments in the initial response candidate data and the text box recognition results in the image sample data to obtain a positional consistency score between the text fragments and the text box recognition results. The intersection-union ratio data and the positional consistency score are normalized to obtain the evidence consistency score; the weight ratio of the positional consistency score is positively correlated with the number of text boxes represented by the text box recognition result.

[0094] When a terminal device performs evidence consistency evaluation on initial response candidate data in multimodal dialogue sample data, it can first calculate the intersection-union ratio (IU / R) of the visual evidence region and the visual attention region corresponding to the initial response candidate data to obtain IU / R data. Then, it performs positional consistency evaluation on the text fragments in the initial response candidate data and the text box recognition results in the image sample data to obtain a positional consistency score between the text fragments and the text box recognition results. Next, the terminal device further normalizes the calculated IU / R data and the evaluated positional consistency score to obtain the evidence consistency score. In this normalization process, the weight of the positional consistency score is positively correlated with the number of text boxes represented by the text box recognition results.

[0095] For example, the terminal device may use the following formula to determine the evidence consistency score e(I, q, r) of the initial response candidate data r to measure the degree of alignment between the initial response candidate data r and the visual evidence in the image sample data I.

[0096] .

[0097] in, The visual evidence regions corresponding to key entities / digits / phrases extracted from the initial response candidate data r. This refers to the visually relevant region (e.g., a heatmap) of image sample data I when the model answers question q. (.) Representation calculation The intersection-union ratio or weighted overlap between Text(r) and CAM(q, I) is defined as follows: Text(r) is a text fragment (such as numbers or proper nouns) extracted from the initial response candidate data r; OCR(I) is the text box recognition result in the image sample data I, that is, all OCR text boxes recognized from the image sample data I; Align(.) is the evidence consistency score, which represents the edit distance or positional consistency between the text boxes in Text(r) and OCR(I); α and β are normalized weights, satisfying α+β=1. For OCR-intensive tasks, β has a higher weight, that is, the weight ratio of the positional consistency score Align(.) is positively correlated with the number of text boxes represented by the text box recognition result.

[0098] In this embodiment, considering that language shortcuts and visual hallucinations are the core challenges in evaluating multimodal large language models (MLLMs), traditional evaluation methods often rely on evidence alignment in only one dimension, such as focusing only on visual localization (based on CAM and IoU) or only on textual facts (based on OCR matching). This embodiment calculates the aforementioned evidence consistency score to identify language shortcut samples that are linguistically reasonable but lack clear visual evidence support through this comprehensive and quantifiable index. When the evidence consistency score is low, it indicates that the model's answer is linguistically reasonable, but the key information it relies on (such as numbers or entities) cannot be clearly supported by visual evidence in the image, or the model did not pay attention to the correct visual area when generating the answer. Therefore, the evidence consistency score is the core index for identifying language shortcut samples.

[0099] Furthermore, this embodiment can use the evidence consistency score as a unified and adjustable metric to comprehensively and flexibly evaluate the model's dependence on complex visual evidence. This allows for flexible adaptation to the different task types' emphasis on visual and textual evidence. For example, in OCR-intensive tasks, the weight of the positional consistency score can be increased. This allows for a more accurate assessment of the dependence of multimodal large language models on key textual information in image sample data.

[0100] Please refer to Figure 4 , Figure 4 for Figure 2 A detailed flowchart of step S203.

[0101] like Figure 4 As shown, in some embodiments, step S203 above: constructing training sample data based on the visual dependency score and the evidence consistency score may include steps S401 and S402 as shown below.

[0102] Step S401: Generate sampling weights based on the visual dependence score, the evidence consistency score, and preset stage-adjustable parameters.

[0103] The preset, adjustable parameters for each stage can be a stage scheduling function and a stage-learnable or preset weight vector. The stage scheduling function balances exploration and exploitation. In the early stages (when t is small), the stage scheduling function tends to select samples with high uncertainty to improve diversity and coverage. In the mid-to-late stages (when t is large), the stage scheduling function tends to select samples with low uncertainty (the model has converged but still has errors), combining high SV and low EC to focus training on the most difficult strong visual samples. The weight vector is used to emphasize different components of s at different training stages (e.g., in the early stages, it might emphasize...). Later emphasis ).

[0104] When constructing training sample data based on visual dependency scores and evidence consistency scores of multimodal dialogue sample data, terminal devices can first generate sampling weights based on the visual dependency scores, evidence consistency scores, and preset adjustable parameters for each stage.

[0105] For example, the terminal device can use the formula shown below, based on the visual dependency score s, the evidence consistency score e, and preset phased adjustable parameters. The sampling weight wt (I, q) is calculated using gt(d).

[0106] .

[0107] Where σ(.) is the activation function Sigmoid, used to convert the linear combination u s is normalized to (0, 1). It is a phased, learnable, or pre-defined weight vector. is the average EC score of the sample in the current training phase, used to identify weak samples for which the model has not yet grasped the evidence, d is the model's uncertainty about the sample (such as the Top-2 probability difference or entropy), and gt(d) is the stage scheduling function.

[0108] Step S402: Perform dynamic sampling processing based on the sampling weights to obtain training sample data.

[0109] After generating the sampling weights, the terminal device can perform dynamic sampling based on these sampling weights to obtain training sample data.

[0110] In some embodiments, the terminal device can perform weighted stratified sampling based on sampling weights, and at the same time, through a task group weight mechanism, ensure that strong visual task clusters such as OCR&Chart, DocVQA, and counting and positioning have a minimum quota (e.g., not less than 30%) in each batch of the sampled training sample data, thereby preventing the training sample data from being diluted by a large number of low SV samples.

[0111] In some embodiments, step S402 above, performing dynamic sampling processing based on the sampling weights to obtain training sample data, may include the following steps: Obtain training sample batches by performing dynamic sampling based on the sampling weights; Based on the positive response training samples that meet the visual dependence requirements in the batch of training samples, near-sense negative response training samples are constructed. Preference sample pairs are generated based on the positive response training samples and the near-sense negative response training samples; the training sample data includes the preference sample pairs.

[0112] When a terminal device performs dynamic sampling based on sampling weights to construct training sample data, it can first perform dynamic sampling based on the sampling weights to obtain training sample batches. Then, for the positive response training samples that meet the visual dependence requirements (e.g., samples with high visual dependence scores), it can construct near-negative response training samples, thereby generating preference sample pairs based on these positive response training samples and near-negative response training samples. These preference sample pairs are then used as training sample data for the multimodal large language model.

[0113] In some embodiments, the step of “constructing a near-sense negative response training sample based on the positive response training samples that meet the visual dependence requirements in the training sample batch” can be performed by at least one of the following methods.

[0114] Method 1: Add text perturbation to the positive response training samples to obtain near-negative response training samples.

[0115] When constructing near-semantic negative response training samples, terminal devices can add textual perturbations to positive response training samples to obtain near-semantic negative response training samples. For example, while maintaining the semantic template of the positive response training sample r + , key numerical values, entities, colors, or positional words can be replaced (e.g., replacing "2026" with "2025", or "blue" with "red"), thus obtaining near-semantic negative response training samples r that are semantically nearest to the positive response training sample r + . However, the near-negative response training sample r This conflict with the evidence shown in image sample data I, thus forcing multimodal large language models to rely on visual details to perform visual interaction tasks.

[0116] Method 2: Delete the visual evidence in the positive response training samples to obtain near-sense negative response training samples.

[0117] When constructing near-sense negative response training samples, terminal devices can also remove visual evidence from positive response training samples to obtain near-sense negative response training samples. For example, partial occlusion (such as covering key OCR text boxes or chart areas) can be applied to image sample data I, or similar layouts can be retrieved across databases (such as replacing them with charts that have similar structures but different data) to obtain near-sense negative response training samples r. In this way, near-sense negative response training samples r can be made While seemingly linguistically plausible, its EC score was significantly lower due to the lack of or conflict of key visual evidence.

[0118] Method 3: Reverse inference is performed on the positive response training samples to obtain near-synonym negative response training samples.

[0119] When constructing near-synonym negative response training samples, the terminal device may also perform reverse inference on the positive response training samples to obtain near-synonym negative response training samples. For example, an auxiliary large language model LLM can be used to reversely generate an incorrect answer that is most likely to be misled by linguistic prior based on the features of the image sample data I and the question data q, so as to obtain the near-synonym negative response training sample r . In this way, it can be ensured that the near-synonym negative response training sample r is the shortcut error most likely to be made by the multimodal large language model, and has the highest catalytic efficiency.

[0120] In some embodiments, the terminal device can generate and catalyze near-synonym negative response samples SNN through an SNN generator, and construct a near-synonym negative response training sample r for the positive response training sample (I, q, r+) , such that the near-synonym negative response training sample r is highly linguistically similar to the positive response training sample r+, but conflicts with the visual evidence of the image training sample data I.

[0121] In some embodiments, after constructing the near-synonym negative response training samples, the terminal device may also adopt a filtering mechanism to perform filtering processing on the near-synonym negative response training samples, so that the filtered near-synonym negative response training samples must satisfy that the evidence consistency score e(I,q,r ) of the near-synonym negative response training samples is significantly lower than the evidence consistency score e(I,q,r+) of the negative response training samples, that is, e(I,q,r ) << e(I,q,r+). In this way, the terminal device can combine the positive response training samples and the filtered near-synonym negative response training samples into a preference sample pair ⟨r+, r ⟩, and send the preference sample pair ⟨r+, r ⟩ as training sample data to an alignment trainer for alignment training of the multimodal large language model.

[0122] In this embodiment, the terminal device adopts an adaptive dynamic sampling strategy to construct training sample data, and then constructs near-synonym negative response training samples, thereby generating preference sample pairs based on the positive response training samples and the near-synonym negative response training samples, and the preference sample pairs will be sent to the alignment trainer for alignment training of the multimodal large language model. In this way, through SV-Aware active sampling and SNN catalysis, this embodiment can fundamentally change the distribution of training data, forcing the multimodal large language model to rely on visual evidence during the training process, thereby systematically suppressing linguistic shortcuts and significantly improving the empirical reliability of the model.

[0123] Please refer to Figure 5 , Figure 5 for Figure 2 A detailed flowchart of step S204.

[0124] like Figure 5 As shown, in some embodiments, the step of “aligning and training the multimodal large language model based on the training sample data” in step S204 above may include steps S501 and S502 as shown below.

[0125] Step S501: Generate alignment preference loss based on the sampling weights and the regularization constraints corresponding to the evidence consistency scores.

[0126] When a terminal device performs alignment training on a multimodal large language model based on training sample data, it can generate an alignment preference loss based on the sampling weights generated when the training sample data was previously constructed and the regularization constraints corresponding to the evidence consistency score obtained by performing evidence consistency evaluation on the multimodal dialogue sample data.

[0127] In some embodiments, standard DPO loss It can be represented by the following formula: .

[0128] in, This is the current multimodal large language model (MLLM) strategy, where β is the temperature coefficient. It is a reference model The logarithmic probability difference.

[0129] Then, the terminal device can use the formula shown below to introduce the sampling weight wt and the regularization constraint corresponding to the evidence consistency score EC generated by SV-Aware into the DPO objective function to generate the alignment preference loss, thereby achieving hard example catalysis and evidence alignment.

[0130] .

[0131] Among them, the weighting term wt(I, q) is the sampling weight generated by SV-Aware, which is used to amplify the gradient contribution of hard examples with high SV and low EC, and the evidence consistency regularization term... The regularization constraint corresponding to the Evidence Consistency Score (EC).

[0132] Thus, through Terminal devices can not only enable multimodal large language models to learn preferences through DPO, but also focus on strong visual difficulties through sampling weights wt, and through This explicitly improves the empirical reliability of multimodal large language models in generating multimodal interaction results.

[0133] Step S502: Input the training sample data into the multimodal large language model for alignment training, and optimize the model parameters of the multimodal large language model based on the alignment preference loss.

[0134] After the terminal device constructs the training sample data and generates the alignment preference loss, it can input the training sample data into the multimodal large language model for alignment training. During the alignment training process of the multimodal large language model, the model parameters are optimized based on the alignment preference loss, thereby obtaining the trained multimodal large language model.

[0135] For example, such as Figure 6 As shown, when the terminal device performs alignment training on the multimodal large language model MLLM based on SV-Aware sampling and SNN catalysis, it extracts samples from the sample pool in the t-th training iteration. That is, it extracts the basic samples to be processed in this iteration from the pre-built multimodal sample pool used for MLLM alignment training, so as to provide input for subsequent scoring calculation.

[0136] In the core SV-Aware sampling process, the SV score is first calculated for each sample, and the EC score is simultaneously calculated for the same sample. Next, the sampling weights are calculated by fusing the SV and EC scores of the same sample to generate a sampling weight for each sample. This provides a quantitative basis for subsequent sampling operations, completing the core weight allocation step of SV-Aware sampling. Then, weighted stratified sampling is performed based on the sampling weights of each sample, prioritizing high-weight samples and filtering low-weight samples to complete the sample selection for this iteration, achieving priority selection of high-value samples based on SV awareness. Finally, a batch acquisition operation is performed to organize the effective samples after weighted stratified sampling according to a preset training batch specification, forming the training batch samples for this iteration and providing standardized input for subsequent training stages.

[0137] In the core SNN catalysis step (negative response construction and filtering), a high-SV sample screening operation is performed. For samples within the current training batch, a threshold is determined based on the SV score, and samples meeting the high-SV standard are selected. Only these samples are sent to the subsequent SNN catalysis step. Next, the SNN generator constructs negative responses. For the selected high-SV samples, the SNN generator generates corresponding negative response samples based on the sample's original input and the baseline positive response, completing the core negative example construction operation of SNN catalysis. Afterwards, a filtering operation is performed to quality-filter the negative responses output by the SNN generator, removing invalid and low-quality negative responses and retaining only valid negative responses that meet the training requirements.

[0138] In the DPO alignment training and iterative closed-loop phase, the preference pair formation operation is first performed. Based on the original positive responses of high-SV samples, the corresponding negative responses after filtering are matched one by one to construct preference pair data that conforms to the DPO training specification. Each preference pair contains positive and negative samples corresponding to the same input. Next, the weighted DPO loss calculation operation is performed. Based on the constructed preference pairs and the sampling weights of the corresponding samples, the weighted DPO loss is calculated. The contribution of different samples to the loss function is adjusted by the sampling weights to strengthen the guiding role of high-SV samples in model alignment. Then, the MLLM parameter optimization operation is performed. Based on the calculated weighted DPO loss, the parameters of the multimodal large language model are updated and optimized through the backpropagation algorithm, completing the core operation of model alignment training in this iteration. After that, the t update operation is performed to update the number of training iterations, completing the complete training iteration process of this round. Finally, the monitoring panel update operation is performed to synchronize the training-related data of this iteration (including score distribution, sampling weights, loss values, parameter update status, etc.) to the monitoring panel, completing the full loop of this iteration.

[0139] Finally, the process returns to the step of extracting samples from the sample pool, starting the next round of training iterations, until the preset training stopping condition is reached.

[0140] Next, a complete embodiment of the interaction method based on a multimodal large language model provided in this application is presented.

[0141] Please refer to Figure 7 , Figure 7 This is a schematic diagram of the technical architecture involved in a complete embodiment of the interaction method based on a multimodal large language model provided in this application.

[0142] like Figure 7As shown, in some embodiments, the terminal device can employ an SV-EC-driven training data routing system to dynamically select samples and construct hard examples during the alignment training (such as DPO) of a multimodal large language model (MLLM). This system uses image I, question I, instruction q, and initial response candidates... The MLLM is trained using the following functional modules in sequence: SV scorer (including positive response r+), OCR results, and layout structure (optional). These modules include the SV scorer, EC evaluator, SV-Aware sampler, SNN generator, alignment trainer, and monitoring panel. Specifically, the SV scorer calculates the SV score vector s(I, q) ∈ Rk for the input (I, q); the EC evaluator calculates the evidence consistency score e(I, q, r) ∈ [0, 1] for the response r; and the SV-Aware sampler uses a dynamic strategy... (s, e) generates the training batch index; the SNN generator constructs a near-sense negative response r for the selected high-SV samples. Forming preference pairs <r+, r >; Alignment trainer optimizes MLLM under weighted DPO1SFT objective The monitoring panel tracks the SV distribution drift and EC boost curve online for closed-loop parameter tuning and training auditing.

[0143] In this embodiment, a data-side training strategy for the post-training stage of the Multimodal Language Model (MLLM) is adopted to enhance the perception and reasoning capabilities of the MLLM model in tasks with high visual dependence, based on strong visual active sampling and near-semantic difficult example catalysis. Specifically, a strong visual scoring vector (SV vector) s is constructed for image-text question answering and multimodal dialogue samples to quantify the sample's dependence on visual information. This is combined with indicators such as OCR density, graph and geometric complexity, instance occlusion, counting, and page entropy, and integrated with the evidence consistency score (EC) to form a joint metric SV-EC, used to evaluate the visual dependence and evidence consistency of the samples. Furthermore, an adaptive dynamic sampling strategy is introduced during the training process of the MLLM model. Initially, the focus was on sample diversity and coverage to promote multimodal generalization. Later, the emphasis shifted to sampling difficult examples with high visual complexity (High SV) and low evidence consistency (Low EC), thus guiding the MLLM model from linguistic shortcuts to reasoning heavily reliant on visual evidence. Furthermore, to enhance visual sensitivity, a semantic nearest neighbor perturbation generator (SNN generator) was used to generate near-synonymous negative response samples that were textually plausible but visually inconsistent. This was achieved through evidence deletion, synonym replacement of key entities, and cross-text / image layout replacement. and the positive and negative responses to < By combining the DPO optimization objective with the optimization level, the model's dependence on language priors is constrained.

[0144] Thus, this embodiment transforms the empirical rule that strong visual tasks benefit more into a measurable and schedulable data construction and training strategy. It can significantly improve the performance of MLLM models in OCR, ChartQA, Document VQA, and counting and localization VQA tasks without modifying the model structure, and effectively suppress language snooping in MLLM models when performing strong visual tasks, achieving a robust transition from language-driven to vision-driven.

[0145] Please refer to Figures 8 to 10 This embodiment, through data-side innovation, systematically suppresses language shortcuts without increasing the complexity of the MLLM model. Specifically, through the combined effect of SNN catalysis and EC regularization, the model is forced to rely on visual evidence when generating answers. In scenarios involving evidence occlusion and cross-image substitution, the error rate of the MLLM model in guessing answers can be reduced by more than 15%. Furthermore, SV-Aware sampling focuses training on high-value difficult examples, achieving an absolute accuracy improvement of +2% to +5% on visually demanding tasks such as OCR & Chart and DocVQA. In addition, this embodiment, through an active sampling mechanism, ensures that each training sample has high information content, achieving near-performance of conventional 40K sample training on a small training set of 3K–10K, while reducing training resource consumption by 70%. Furthermore, this embodiment can be embedded into the SFT1DPO process as training middleware without modifying the network structure of the MLLM model, thus ensuring compatibility with mainstream MLLM architectures such as LLaVA and MiniGPT-4. The SV1EC curve can be tracked in real-time on the monitoring panel as a core metric, providing clear and interpretable indicators for data governance and training auditing, facilitating online parameter tuning. Finally, this embodiment improves the empirical reliability of the MLLM model in scenarios such as table reading, label recognition, and quantity estimation, making it highly valuable for embodied intelligence and industrial automation applications requiring high-reliability visual perception.

[0146] Please refer to Figure 11 This application also provides an interactive system based on a multimodal large language model. The interactive system based on a multimodal large language model provided in this application can realize the above-mentioned interactive method based on a multimodal large language model.

[0147] like Figure 11 As shown in the embodiments of this application, the interactive system based on a multimodal large language model may include: The acquisition module is used to acquire image data and query data associated with the image data; The model interaction module is used to perform visual interaction task processing on the image data and the question data based on a preset multimodal large language model to obtain the multimodal interaction result of the question data; the multimodal large language model is obtained by alignment training based on training sample data driven by visual dependence and evidence consistency, the visual dependence is used to characterize the degree of dependence of the visual interaction task on visual evidence, and the evidence consistency is used to characterize the degree of alignment between the multimodal interaction result and the visual evidence.

[0148] In some embodiments, the interactive system based on a multimodal large language model provided in this application may further include: The model training module is used to acquire multimodal dialogue sample data; perform visual dependency assessment on the multimodal dialogue sample data to obtain a visual dependency score; and perform evidence consistency assessment on the multimodal dialogue sample data to obtain an evidence consistency score; construct training sample data based on the visual dependency score and the evidence consistency score; and perform alignment training on the multimodal large language model based on the training sample data to obtain the trained multimodal large language model.

[0149] In some embodiments, the multimodal dialogue sample data includes: image sample data and question sample data associated with the image sample data. The model training module is further configured to perform visual dependency evaluation processing on the image sample data and the question sample data based on a preset multidimensional visual dependency quantification index to obtain a multidimensional visual dependency score vector; the visual dependency score includes the multidimensional visual dependency score vector.

[0150] In some embodiments, the model training module is further configured to extract visual features from the image data, and to convert the question sample data into semantic features; to perform fusion processing on the visual features and the semantic features to obtain joint visual-semantic features; to perform visual dependency evaluation processing on the joint visual-semantic features in parallel based on a preset multi-task regression head to obtain a multi-dimensional visual dependency score vector output by the multi-task regression head; the multi-task regression head corresponds one-to-one with the multi-dimensional visual dependency quantification index.

[0151] In some embodiments, the multimodal dialogue sample data further includes: initial response candidate data of the question sample data. In some embodiments, the model training module is further configured to calculate the intersection-union ratio (IUGR) of the visual evidence region and the visual attention region corresponding to the initial response candidate data to obtain IUGR data between the visual evidence region and the visual attention region; perform positional consistency evaluation processing on the text fragments in the initial response candidate data and the text box recognition results in the image sample data to obtain a positional consistency score between the text fragments and the text box recognition results; normalize the IUGR data and the positional consistency score to obtain an evidence consistency score; the weight ratio of the positional consistency score is positively correlated with the number of text boxes represented by the text box recognition results.

[0152] In some embodiments, the model training module is further configured to generate sampling weights based on the visual dependency score, the evidence consistency score, and preset stage-adjustable parameters; and perform dynamic sampling processing based on the sampling weights to obtain training sample data.

[0153] In some embodiments, the model training module is further configured to obtain a batch of training samples obtained by performing dynamic sampling based on the sampling weights; construct near-negative response training samples based on the positive response training samples in the batch of training samples that meet the visual dependence requirements; generate preference sample pairs based on the positive response training samples and the near-negative response training samples; and the training sample data includes the preference sample pairs.

[0154] In some embodiments, the model training module is further configured to add text perturbations to the positive response training samples to obtain near-negative response training samples; delete visual evidence from the positive response training samples to obtain near-negative response training samples; and perform reverse reasoning on the positive response training samples to obtain near-negative response training samples.

[0155] In some embodiments, the model training module is further configured to generate an alignment preference loss based on the sampling weights and the regularization constraints corresponding to the evidence consistency score; input the training sample data into the multimodal large language model for alignment training; and optimize the model parameters of the multimodal large language model based on the alignment preference loss.

[0156] It should be noted that the specific implementation of the interactive system based on the multimodal large language model provided in this application is basically the same as the specific implementation of the interactive method based on the multimodal large language model described above, and will not be repeated here.

[0157] Please see Figure 12This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described interaction method based on a multimodal large language model.

[0158] In some embodiments, the electronic device may be any intelligent terminal such as an intelligent agent, an in-vehicle terminal, an in-vehicle hardware platform (e.g., an in-vehicle computer), a tablet computer, a smartphone, and a wearable device; or, the electronic device may be a vehicle / robot including a memory and a processor.

[0159] like Figure 12 As shown, the electronic device provided in this application embodiment may include: The processor 1201 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1202 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1202 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1202 and is called and executed by the processor 1201 to execute the interaction method based on a multimodal large language model according to the embodiments of this application. The input / output interface 1203 is used to implement information input and output; The communication interface 1204 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1205 transmits information between various components of the device (e.g., processor 1201, memory 1202, input / output interface 1203, and communication interface 1204); The processor 1201, memory 1202, input / output interface 1203 and communication interface 1204 are connected to each other within the device via bus 1205.

[0160] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described interaction method based on a multimodal large language model.

[0161] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0162] This application also provides a computer program product, including a computer program. The steps implemented by the computer program when executed by a processor are basically the same as those in the specific embodiments of the interaction method based on the multimodal large language model described above, and will not be repeated here.

[0163] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of this application, and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0164] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0165] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0166] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0167] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0168] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding factor is divided by the following factor, or that the related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0169] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0170] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0171] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0172] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0173] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. An interaction method based on a multimodal large language model, characterized in that, The method includes: Acquire image data and query data associated with the image data; The image data and the question data are processed using a pre-defined multimodal large language model to obtain the multimodal interaction results of the question data. The multimodal large language model is obtained by aligning and training the multimodal large language model based on training sample data, including: Acquire multimodal dialogue sample data; the multimodal dialogue sample data includes: image sample data and question sample data associated with the image sample data; Extract visual features from the image sample data, and convert the question sample data into semantic features; The visual features and the semantic features are fused to obtain joint visual-semantic features; The visual dependence evaluation process is performed on the visual semantic joint features in parallel based on a preset multi-task regression head to obtain a multi-dimensional visual dependence score vector output by the multi-task regression head, and the multi-dimensional visual dependence score vector is used as the visual dependence score; the multi-task regression head corresponds one-to-one with the multi-dimensional visual dependence quantification index. The evidence consistency assessment is performed on the multimodal dialogue sample data to obtain an evidence consistency score; Training sample data is constructed based on the visual dependence score and the evidence consistency score; The multimodal large language model is aligned and trained based on the training sample data to obtain the trained multimodal large language model.

2. The method according to claim 1, characterized in that, The multimodal dialogue sample data further includes: initial response candidate data of the question sample data; the evidence consistency evaluation process for the multimodal dialogue sample data to obtain an evidence consistency score includes: The intersection-union ratio (IU) of the visual evidence region and the visual attention region corresponding to the initial response candidate data is calculated to obtain the IU data between the visual evidence region and the visual attention region. A positional consistency evaluation process is performed on the text fragments in the initial response candidate data and the text box recognition results in the image sample data to obtain a positional consistency score between the text fragments and the text box recognition results. The intersection-union ratio data and the positional consistency score are normalized to obtain the evidence consistency score; the weight ratio of the positional consistency score is positively correlated with the number of text boxes represented by the text box recognition result.

3. The method according to claim 1, characterized in that, The construction of training sample data based on the visual dependency score and the evidence consistency score includes: Based on the visual dependence score, the evidence consistency score, and the preset phased adjustable parameters, sampling weights are generated; Dynamic sampling processing is performed based on the sampling weights to obtain training sample data.

4. The method according to claim 3, characterized in that, The step of performing dynamic sampling processing based on the sampling weights to obtain training sample data includes: Obtain training sample batches by performing dynamic sampling based on the sampling weights; Based on the positive response training samples that meet the visual dependence requirements in the training sample batch, near-sense negative response training samples are constructed. Preference sample pairs are generated based on the positive response training samples and the near-sense negative response training samples; the training sample data includes the preference sample pairs.

5. The method according to claim 4, characterized in that, The near-negative response training samples constructed based on the positive response training samples that meet the visual dependence requirement in the batch of training samples include at least one of the following: Text perturbation is added to the positive response training samples to obtain near-negative response training samples; Visual evidence is removed from the positive response training samples to obtain near-sense negative response training samples. By performing reverse reasoning on the positive response training samples, near-sense negative response training samples are obtained.

6. The method according to claim 3, characterized in that, The alignment training of the multimodal large language model based on the training sample data includes: Based on the sampling weights and the regularization constraints corresponding to the evidence consistency scores, an alignment preference loss is generated. The training sample data is input into the multimodal large language model for alignment training, and the model parameters of the multimodal large language model are optimized based on the alignment preference loss.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the interaction method based on a multimodal large language model as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the interaction method based on a multimodal large language model as described in any one of claims 1 to 6.

9. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the interaction method based on a multimodal large language model as described in any one of claims 1 to 6.

Images