Multi-modal large model optimization alignment method and system based on uncertainty estimation

Abstract

The application discloses a multi-modal large model optimization alignment method and system based on uncertainty estimation, and the method comprises the following steps: (1) overall uncertainty estimation of data: calculating the overall uncertainty of all four-tuples in the training data set; (2) batch uncertainty measurement: for each individual in a given batch sample, the uncertainty thereof is calculated, and abnormal samples are removed through a dynamic selection mechanism to obtain smoothed batch uncertainty; (3) model optimization adjustment. Firstly, uncertainty estimation is performed on the input preference data pair based on the model output, wherein the uncertainty of the difficult sample pair before is low. Based on the model optimization mode, the measurement result of the multi-modal large model is integrated into the multi-modal large model preference optimization process, and higher weight is allocated to the sample pair with high uncertainty in the optimization process, so that attention is paid to the difficult sample, thereby realizing robust optimization.

Description

Technical Field

[0001] The present application relates to the technical field of large language models, and more specifically to a multimodal large model optimization alignment method and system based on uncertainty estimation. Background Art

[0002] With the rapid development of large AI models, multi-modal large language models (MLLMs) have achieved remarkable success in multimodal understanding and other areas. These models can perform tasks such as multi-turn dialogue, visual question answering, and visual reasoning, greatly promoting the development of the multimodal field.

[0003] However, in their application, large multimodal models face a key challenge: ensuring that the model output is consistent with the true content of the image, that is, how to address the hallucination problem of large multimodal models. To address this issue, researchers have proposed a model alignment strategy, which involves incorporating human preferences into data construction and further fine-tuning the model by optimizing the alignment, so that the model generates responses that are more consistent with the image content.

[0004] Direct Preference Optimization (DPO), a typical approach, directly utilizes preference data to guide model learning, simplifying the model training process and improving the efficiency of model optimization and alignment. Its core idea is to dynamically adjust the model's output probability by comparing data that humans prefer with data that they dislike, making the model more likely to generate desired responses.

[0005] However, large multimodal models face the problem of inconsistent quality between preferred and unpreferred data pairs during the optimization and alignment process. Low-quality data pairs are highly differentiated and easier to fit, while high-quality data pairs are less differentiated and more difficult to fit. If the model is overly influenced by simple, low-quality data, it will be unable to accurately capture and respond to new scenarios, resulting in poor performance in difficult scenarios. Summary of the Invention

[0006] In this embodiment, a multimodal large model optimization alignment method, system, electronic device and storage medium based on uncertainty estimation are provided to solve the problem of inconsistent quality of preferred and non-preferred data pairs in the optimization alignment process in related technologies.

[0007] In a first aspect, an embodiment of the present invention provides a multimodal large model optimization alignment method based on uncertainty estimation, the multimodal large model optimization alignment method based on uncertainty estimation comprising:

[0008] (1) Data overall uncertainty estimation: Calculate the overall uncertainty of all quadruple in the training dataset;

[0009] (2) Batch uncertainty measurement: For each individual in a given batch sample, its uncertainty is calculated, and abnormal samples are eliminated through a dynamic selection mechanism to obtain the smoothed batch uncertainty;

[0010] (3) Model optimization and adjustment: Dynamically adjust the temperature coefficient in the optimization function according to the batch uncertainty, generate an improved loss function, and update the model parameters based on the loss function.

[0011] Preferably, the overall uncertainty is expressed as:

[0012] ;

[0013] in, Represents a sample, is the text input for the multimodal large model, is the image input of the multimodal large model, is the expected output of the multimodal large model, is the output of the undesirable multimodal large model, is the reward function, and N represents the total number of quadruples in the dataset.

[0014] Preferably, the reward function expression is as follows:

[0015] ;

[0016] in, For the multimodal large language model to be optimized, is the baseline model.

[0017] Preferably, the specific steps of batch uncertainty measurement include:

[0018] Calculate batch samples The uncertainty of each individual ;

[0019] Setting thresholds , according to the individual uncertainty and or The comparison results determine the sample rejection rate ,in, is the overall uncertainty;

[0020] Eliminate rejected samples, calculate the batch uncertainty, and obtain the smoothed uncertainty by the ratio of the batch uncertainty to the overall uncertainty.

[0021] Preferably, the determination and adjustment of the sample rejection rate is as follows:

[0022] If the individual uncertainty satisfy or , then the rejection rate of the sample Set to 1, otherwise the rejection rate is set to 0.

[0023] Preferably, the temperature coefficient is adjusted as follows:

[0024] ;

[0025] in, is the temperature coefficient, is the adjusted temperature coefficient, is the uncertainty after smoothing.

[0026] Preferably, the improved loss function expression is:

[0027] ;

[0028] in, represents the parameters of the multimodal large model to be optimized, is the adjusted temperature coefficient, Q is the text input of the multimodal large model, I is the image input of the multimodal large model, is the expected output of human preference, Output that is not preferred by humans, is the multimodal large model to be optimized, is the baseline model, is the Sigmoid function.

[0029] Compared with the prior art, the multimodal large model optimization alignment method based on uncertainty estimation of the present invention has the following beneficial effects:

[0030] To address the hallucination problem of current large multimodal models, we propose an optimized alignment scheme based on uncertainty estimation. This systematically analyzes the quality of aligned sample pairs, identifies difficult samples, and improves the model's responsiveness to these challenges. By incorporating a difficult sample selection mechanism into the model optimization process and adaptively adjusting the temperature coefficient of DPO training, we improve the model's robustness.

[0031] This method first estimates the uncertainty of the input preference data pairs based on the model output, where the uncertainty of the difficult sample pairs is low. Then, based on the model optimization method, the measurement results of the multimodal large model are integrated into the preference optimization process of the multimodal large model. By assigning higher weights to sample pairs with high uncertainty during the optimization process, the method focuses on difficult samples and achieves robust optimization.

[0032] In a second aspect, an embodiment of the present invention provides a multimodal large model optimization alignment system based on uncertainty estimation, comprising:

[0033] The data overall uncertainty estimation module is configured to: calculate the overall uncertainty of all quadruple groups in the training data set;

[0034] The batch uncertainty measurement module is configured to: calculate the uncertainty of each individual in a given batch sample, and remove abnormal samples through a dynamic selection mechanism to obtain the smoothed batch uncertainty;

[0035] The model optimization and adjustment module is configured to dynamically adjust the temperature coefficient in the optimization function according to the batch uncertainty, generate an improved loss function, and update the model parameters based on the loss function.

[0036] In a third aspect, an embodiment of the present invention provides an electronic device comprising a processor, a communication interface, a memory and a bus, wherein the processor, the communication interface and the memory communicate with each other through the bus, and the processor can call logic instructions in the memory to execute the steps of the method provided in the first aspect.

[0037] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal large model optimization alignment method based on uncertainty estimation as described in the first aspect.

[0038] Compared with the prior art, the beneficial effects of the multimodal large model optimization alignment system, electronic device and storage medium based on uncertainty estimation of the present invention are the same as the multimodal large model optimization alignment method based on uncertainty estimation described in the first aspect, so they will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 This is a flow chart of a multimodal large model optimization alignment method based on uncertainty estimation in an embodiment of the present invention;

[0041] Figure 2 This is a structural block diagram of a multimodal large model optimization alignment system based on uncertainty estimation in an embodiment of the present invention;

[0042] Figure 3 2 is a structural block diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to more clearly understand the purpose, technical solutions and advantages of this application, this application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0044] Unless otherwise defined, technical or scientific terms used in this application shall have the ordinary meanings as understood by persons of ordinary skill in the art to which this application belongs. The terms "a," "an," "the," "these," and similar expressions in this application do not denote limitations on quantity and may be singular or plural. The terms "comprise," "include," "have," and any variations thereof, as used in this application, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include unlisted steps or modules (units) or other steps or modules (units) inherent to the process, method, product, or device. The terms "connected," "connected," "coupled," and similar expressions used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. As used in this application, "plurality" means two or more. "And / or" describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone; A and B exist simultaneously; or B exists alone. Generally, the character " / " indicates that the objects in the preceding and following relationship are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0045] In an embodiment of the present invention, a multimodal large model optimization alignment method based on uncertainty estimation is provided. Figure 1 This is a flow chart of the multimodal large model optimization alignment method based on uncertainty estimation of the present invention, as shown in FIG. Figure 1 As shown, the process includes the following steps:

[0046] S100, data overall uncertainty estimation: calculate the overall uncertainty of all quadruples in the training data set;

[0047] Specifically, before performing uncertainty estimation, in order to prevent bias, the uncertainty in the statistical data set is first calculated. For all the quadruple in the training data set, , calculate the overall uncertainty of the data , expressed as:

[0048] ;

[0049] in, Represents a sample, is the text input for the multimodal large model, is the image input of the multimodal large model, is the expected output of the multimodal large model, is the output of the undesirable multimodal large model, is the reward function, and N represents the total number of quadruples in the dataset.

[0050] Furthermore, the reward function expression is as follows:

[0051] ;

[0052] in, For the multimodal large language model to be optimized, is the baseline model.

[0053] S200, Batch uncertainty measurement: For each individual in a given batch sample, calculate its uncertainty and remove abnormal samples through a dynamic selection mechanism to obtain a smoothed batch uncertainty;

[0054] In this embodiment, the specific steps of batch uncertainty measurement include:

[0055] Calculate batch samples The uncertainty of each individual ;

[0056] Setting thresholds , according to the individual uncertainty and or The comparison results determine the sample rejection rate ,in, is the overall uncertainty;

[0057] Eliminate rejected samples, calculate the batch uncertainty, and obtain the smoothed uncertainty by the ratio of the batch uncertainty to the overall uncertainty.

[0058] The determination of the sample rejection rate is adjusted as follows:

[0059] If the individual uncertainty satisfy or , then the rejection rate of the sample Set to 1, otherwise the rejection rate is set to 0.

[0060] Specifically, during the model optimization process, for a given batch of samples Each individual in , and its uncertainty is expressed as:

[0061] ;

[0062] In the process of model optimization, in order to prevent individuals with too high or too low reward values ​​from affecting the uncertainty statistics of batch samples, a dynamic data selection mechanism is introduced. Specifically, for a given sample And batch size B, set the threshold , rejection rate Expressed as:

[0063] ;

[0064] If the uncertainty of individual samples Above average The rejection rate of samples or below average The rejection rate of samples is Set to 1, otherwise the rejection rate is set to 0.

[0065] Based on individual uncertainty and sample rejection rate, batch samples The uncertainty of is expressed as:

[0066]

[0067] Furthermore, considering the uncertainty of the data as a whole, Uncertainty Smoothing is performed to obtain the smoothed uncertainty, which is expressed as:

[0068] ;

[0069] S300, model optimization adjustment: dynamically adjust the temperature coefficient in the optimization function according to the batch uncertainty, generate an improved loss function, and update the model parameters based on the loss function.

[0070] Based on the above uncertainty measurement method, the dynamic sample selection mechanism is integrated into the model optimization process. Specifically, for a given batch of samples after dynamic selection, the original optimization function is expressed as:

[0071] ;

[0072] in, is the temperature coefficient in the model optimization process. In order to integrate uncertainty measurement into the model optimization process, it is dynamically adjusted according to the uncertainty in the batch data. In this embodiment, the temperature coefficient The adjustment is:

[0073] ;

[0074] in, is the temperature coefficient, is the adjusted temperature coefficient, is the uncertainty after smoothing.

[0075] During the model optimization process, for a given batch of samples, the improved optimization function (loss function) is expressed as:

[0076] ;

[0077] in, represents the parameters of the multimodal large model to be optimized, is the adjusted temperature coefficient, Q is the text input of the multimodal large model, I is the image input of the multimodal large model, is the expected output of human preference, Output that is not preferred by humans, is the multimodal large model to be optimized, is the baseline model, is the Sigmoid function.

[0078] By optimizing a given batch of samples based on an improved optimization function, the model can dynamically capture and optimize difficult samples based on uncertainty, thereby improving the model's ability to respond to difficult scenarios.

[0079] In summary, to address the hallucination problem of current large multimodal models, we propose an optimized alignment scheme based on uncertainty estimation. This systematically analyzes the quality of aligned sample pairs, identifies difficult samples, and improves the model's responsiveness to these challenges. By incorporating a difficult sample selection mechanism into the model optimization process and adaptively adjusting the temperature coefficient of DPO training, we improve the model's robustness.

[0080] This method first estimates the uncertainty of the input preference data pairs based on the model output, where the uncertainty of the difficult sample pairs is low. Then, based on the model optimization method, the measurement results of the multimodal large model are integrated into the preference optimization process of the multimodal large model. By assigning higher weights to sample pairs with high uncertainty during the optimization process, the method focuses on difficult samples and achieves robust optimization.

[0081] The embodiments of the present invention also provide a multimodal large model optimization alignment system based on uncertainty estimation, which is used to implement the above-mentioned method embodiments. The terms "module", "unit", "sub-unit", etc. used below can implement a combination of software and / or hardware for predetermined functions. Although the system described in the following embodiments is preferably implemented in software, implementation in hardware or a combination of software and hardware is also possible and conceivable.

[0082] like Figure 2 As shown, Figure 2 This is a structural block diagram of a multimodal large model optimization alignment system based on uncertainty estimation in the present invention, which includes:

[0083] The data overall uncertainty estimation module 101 is configured to: calculate the overall uncertainty of all quadruple groups in the training data set;

[0084] The batch uncertainty measurement module 102 is configured to: calculate the uncertainty of each individual in a given batch sample, and remove abnormal samples through a dynamic selection mechanism to obtain a smoothed batch uncertainty;

[0085] The model optimization and adjustment module 103 is configured to dynamically adjust the temperature coefficient in the optimization function according to the batch uncertainty, generate an improved loss function, and update the model parameters based on the loss function.

[0086] The beneficial effects of the multimodal large model optimization alignment system based on uncertainty estimation of the present invention are the same as those of the multimodal large model optimization alignment method based on uncertainty estimation described in the first aspect, so they will not be repeated here.

[0087] Figure 3 A structural block diagram of an electronic device provided by an embodiment of the present invention, such as Figure 3 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the following method:

[0088] S100, data overall uncertainty estimation: calculate the overall uncertainty of all quadruples in the training data set;

[0089] S200, Batch uncertainty measurement: For each individual in a given batch sample, calculate its uncertainty and remove abnormal samples through a dynamic selection mechanism to obtain a smoothed batch uncertainty;

[0090] S300, model optimization adjustment: dynamically adjust the temperature coefficient in the optimization function according to the batch uncertainty, generate an improved loss function, and update the model parameters based on the loss function.

[0091] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0092] An embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method provided in the above embodiments is implemented.

[0093] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus the necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or certain parts of the embodiment.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A multimodal large model optimization alignment method based on uncertainty estimation is used to improve the consistency between the multimodal large model output and image content and solve the model hallucination problem. It is characterized by: The method comprises: (1) Data overall uncertainty estimation: Calculate the overall uncertainty of all quadruple in the training dataset; (2) Batch uncertainty measurement: For each individual in a given batch sample, its uncertainty is calculated, and abnormal samples are eliminated through a dynamic selection mechanism to obtain the smoothed batch uncertainty; (3) Model optimization and adjustment: Dynamically adjust the temperature coefficient in the direct preference optimization (DPO) loss function based on the smoothed batch uncertainty to generate an improved loss function, and update the parameters of the multimodal large model based on the loss function to improve the model's responsiveness to difficult samples and optimization robustness; The temperature coefficient is adjusted as follows: ； in, is the temperature coefficient, is the adjusted temperature coefficient, is the uncertainty after smoothing; The improved loss function expression is: ； in, represents the parameters of the multimodal large model to be optimized, is the adjusted temperature coefficient, Q is the text input of the multimodal large model, I is the image input of the multimodal large model, is the expected output of human preference, Output that is not preferred by humans, is the multimodal large model to be optimized, is the baseline model, is the Sigmoid function.

2. The multimodal large model optimization alignment method based on uncertainty estimation according to claim 1 is characterized in that: The overall uncertainty is expressed as: ； in, Represents a sample, is the text input for the multimodal large model, is the image input of the multimodal large model, is the expected output of the multimodal large model, is the output of the undesirable multimodal large model, is the reward function, and N represents the total number of quadruples in the dataset.

3. The multimodal large model optimization alignment method based on uncertainty estimation according to claim 2 is characterized in that: The reward function expression is as follows: ； in, For the multimodal large language model to be optimized, is the baseline model.

4. The multimodal large model optimization alignment method based on uncertainty estimation according to claim 1 is characterized in that: The specific steps of batch uncertainty measurement include: Calculate batch samples The uncertainty of each individual ; Setting thresholds , according to the individual uncertainty and or The comparison results determine the sample rejection rate ,in, is the overall uncertainty; Eliminate rejected samples, calculate the batch uncertainty, and obtain the smoothed uncertainty by the ratio of the batch uncertainty to the overall uncertainty.

5. The multimodal large model optimization alignment method based on uncertainty estimation according to claim 4 is characterized in that: The determination and adjustment of the sample rejection rate is: If the individual uncertainty satisfy or , then the rejection rate of the sample Set to 1, otherwise the rejection rate is set to 0.

6. A multimodal large model optimization alignment system based on uncertainty estimation, characterized by: include: The data overall uncertainty estimation module is configured to: calculate the overall uncertainty of all quadruple groups in the training data set; The batch uncertainty measurement module is configured to: calculate the uncertainty of each individual in a given batch sample, and remove abnormal samples through a dynamic selection mechanism to obtain the smoothed batch uncertainty; a model optimization and adjustment module configured to dynamically adjust the temperature coefficient in the direct preference optimization (DPO) loss function according to the smoothed batch uncertainty to generate an improved loss function, and update the parameters of the multimodal large model based on the loss function to improve the model's responsiveness to difficult samples and optimization robustness; The temperature coefficient is adjusted as follows: ； in, is the temperature coefficient, is the adjusted temperature coefficient, is the uncertainty after smoothing; The improved loss function expression is: ； in, represents the parameters of the multimodal large model to be optimized, is the adjusted temperature coefficient, Q is the text input of the multimodal large model, I is the image input of the multimodal large model, is the expected output of human preference, Output that is not preferred by humans, is the multimodal large model to be optimized, is the baseline model, is the Sigmoid function.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the multimodal large model optimization alignment method based on uncertainty estimation according to any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multimodal large model optimization alignment method based on uncertainty estimation are implemented as described in any one of claims 1 to 5.

Images