A conditional preference optimization method, system, device and medium for multimodal information processing

By acquiring image and text preference response data pairs through a multimodal large-scale language model, and utilizing an improved conditional preference optimization algorithm, the problem of visual information not being considered in the multimodal model is solved, thereby improving the model's accuracy in representing human preferences and reducing the risk of hallucination generation.

CN118569388BActive Publication Date: 2024-11-22GLORYVIEW TECH INC
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411036209.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-22
Estimated Expiration
2044-07-31

AI Technical Summary

Technical Problem

Existing multimodal large-scale language models fail to adequately consider non-linguistic information such as visual information during preference optimization, resulting in inaccurate alignment of preference decisions and potentially exacerbating problems such as hallucination generation.

Method used

By acquiring preference-response data pairs of question-associated images and text through a multimodal large-scale language model, and utilizing improved conditional preference optimization algorithms, including direct preference optimization, multi-objective preference optimization, and self-attention reward strategies, the self-attention reward strategy is introduced during the optimization process to ensure that the reward for the best response is not reduced, thereby improving the model's accuracy in understanding human preferences.

Benefits of technology

It improves the accuracy of large multimodal language models in processing multimodal inputs, avoids the problem of relying solely on language information while ignoring image information, reduces the risk of hallucination generation, and ensures the quality of optimal responses.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118569388B_ABST
    Figure CN118569388B_ABST
Patent Text Reader

Abstract

The present invention relates to the technical field of large language models, and discloses a conditional preference optimization method, system, device and medium for multimodal information processing, including: obtaining a preference answer data pair corresponding to a question-associated image and a preference answer data pair corresponding to a question text through a multimodal large language model according to user input data; obtaining the optimal answer corresponding to the user input data by using an improved conditional preference optimization algorithm according to the user input data and the preference answer data pair corresponding to the question-associated image and the preference answer data pair corresponding to the question text, wherein the improved conditional preference optimization algorithm includes a direct preference optimization algorithm, a multi-objective preference optimization algorithm and a self-attention reward strategy. The present invention maintains the quality of the preferred response, improves the multimodal large language model when processing multimodal inputs such as images and texts, and can more accurately reflect human preferences, while avoiding the problem of relying only on language information and ignoring image information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of large language models, and in particular to a conditional preference optimization method, system, device and medium for multimodal information processing. Background Art

[0002] With the rapid development of artificial intelligence technology, multimodal large language models (MLLMs) have become a hot spot for cross-domain applications. They can understand, generate and combine multiple modal information such as text and images, and are widely used in virtual assistants, content creation, intelligent recommendation systems and other fields. However, how to effectively align the preferences of these models to ensure that the content they generate is both accurate and in line with human values ​​and social norms has become a key issue that needs to be solved urgently.

[0003] At present, Direct Preference Optimization (DPO) has been successfully applied to large language models of a single language modality as an effective means. By learning directly from preference data, the consistency of the model with human preferences is improved. However, when DPO is extended to multimodal scenarios, it encounters significant challenges. Studies have shown that simply replacing text preference data with multimodal preference data does not always lead to consistent performance improvements, but may exacerbate problems such as hallucination generation. This is mainly because the model often ignores the conditions of non-language modalities such as images in the process of multimodal preference optimization, resulting in the learning of "unconditional preference", that is, preference decisions do not fully consider non-language information such as vision.

[0004] Therefore, there is an urgent need for a conditional preference optimization method for multimodal information processing. Summary of the invention

[0005] The present invention provides a conditional preference optimization method, system, device and medium for multimodal information processing, which are used to solve the defect that preference decisions of large language models in the current multimodal preference optimization process do not fully consider non-linguistic information such as vision, resulting in inaccurate preference alignment.

[0006] The present invention provides a conditional preference optimization method for multimodal information processing, comprising:

[0007] Acquire user input data, wherein the user input data includes a question-related image and a question text;

[0008] According to the user input data, a preference answer data pair corresponding to the question-related image and a preference answer data pair corresponding to the question text are obtained through a multimodal large language model;

[0009] According to the user input data and the preference answer data pairs corresponding to the question-associated image and the preference answer data pairs corresponding to the question text, an improved conditional preference optimization algorithm is used to obtain the optimal answer corresponding to the user input data, wherein the improved conditional preference optimization algorithm includes a direct preference optimization algorithm, a multi-objective preference optimization algorithm and a self-attention reward strategy.

[0010] According to a conditional preference optimization method for multimodal information processing provided by the present invention, the method obtains a preference answer data pair corresponding to a question-associated image and a preference answer data pair corresponding to a question text through a multimodal large language model based on user input data, including:

[0011] According to the user input data, a target image related to the question text is obtained from the question-related image through a multimodal large language model, and answer data corresponding to the target image is obtained, so as to form a preference answer data pair having a one-to-one relationship with the target image as the preference answer data pair corresponding to the question-related image;

[0012] According to the user input data, the answer data corresponding to the question text is obtained through a multimodal large language model, and a preference answer data pair having a one-to-one relationship with the question text is formed as the preference answer data pair corresponding to the question text.

[0013] According to a conditional preference optimization method for multimodal information processing provided by the present invention, the method acquires a target image related to the question text from the question-related image through a multimodal large language model according to user input data, including:

[0014] According to the user input data, a multimodal large language model is used to use the target detection algorithm to obtain the target image related to the question text from the question-related images;

[0015] or,

[0016] According to user input data, a target image related to the question text is obtained from question-associated images through a multimodal large language model, wherein the multimodal large language model generates a large model for image description.

[0017] According to a conditional preference optimization method for multimodal information processing provided by the present invention, the optimal answer corresponding to the user input data is obtained by using an improved conditional preference optimization algorithm based on the user input data and the preference answer data pair corresponding to the question-associated image and the preference answer data pair corresponding to the question text, wherein the improved conditional preference optimization algorithm includes a direct preference optimization algorithm, a multi-objective preference optimization algorithm and a self-attention reward strategy, including:

[0018] The preference answer data pairs corresponding to the question text are optimized using the direct preference optimization algorithm, and a direct preference optimization loss is obtained in each round of optimization.

[0019] According to the user input data and the preference answer data pairs corresponding to the image associated with the question, the preference answer data pairs corresponding to the image associated with the question are optimized using a multi-objective preference optimization algorithm, and a multi-objective preference optimization loss is obtained in each round of optimization process;

[0020] Using the self-attention reward strategy, a reward anchor is introduced to ensure that the reward of the preferred response remains positive during the optimization process, and a self-attention reward strategy loss is obtained in each round of optimization;

[0021] According to the above steps, multiple rounds of iterative optimization are performed. In each round of optimization, the sum of the direct preference optimization loss, multi-objective preference optimization loss, and self-attention reward strategy loss of that round is used as the total loss of that round, until the total loss meets the preset range, that is, the answer data corresponding to the total loss is used as the optimal answer corresponding to the user input data.

[0022] According to a conditional preference optimization method for multimodal information processing provided by the present invention, the overall objective function of the improved conditional preference optimization algorithm is:

[0023] ,

[0024] In the formula, represents the sum of direct preference optimization loss, multi-objective preference optimization loss, and self-attention reward strategy loss, denotes a direct preference for optimizing loss, represents the multi-objective preference optimization loss, Represents the self-attention reward strategy loss.

[0025] According to a conditional preference optimization method for multimodal information processing provided by the present invention, the objective function of the direct preference optimization algorithm is:

[0026] ,

[0027] In the formula, denotes a direct preference for optimizing loss, represents the sigmoid function used to convert scores into probabilities, Represents the question text of the multimodal large language model in the user input data Generate the best answer The probability of Represents the question text of the multimodal large language model in the user input data Generate non-preferred answers The probability of Represents the problem text of the preset reference strategy / prior distribution over the user input data Generate the best answer The probability of Represents the problem text of the preset reference strategy / prior distribution over the user input data Generate non-preferred answers The probability of represents a hyperparameter used to adjust the degree to which the multimodal large language model deviates from the reference model, represents an anchor value to ensure that the reward for the preferred response is always greater than the anchor value.

[0028] According to a conditional preference optimization method for multimodal information processing provided by the present invention, the objective function of the multi-objective preference optimization algorithm is:

[0029] ,

[0030] In the formula, represents the multi-objective preference optimization loss, represents the sigmoid function used to convert scores into probabilities, Represents a multimodal large language model in the question text and the target image Generate the target image below The probability of Represents a multimodal large language model in the question text Associate image with the original question Generate the original image below The probability of Indicates the preset reference strategy / prior distribution in the question text and the target image Generate the target image below and the target image is not the preferred answer The probability of Indicates the preset reference strategy / prior distribution in the question text Associate image with the original question Generate the original image below and the original image non-preferred answer The probability of represents a hyperparameter used to adjust the degree to which the multimodal large language model deviates from the reference model, represents an anchor value to ensure that the reward for the preferred response is always greater than the anchor value.

[0031] According to a conditional preference optimization method for multimodal information processing provided by the present invention, the joint probability density function of the preset reference strategy / prior distribution is:

[0032] ,

[0033] In the formula, ( ) represent variables and variables The mean vector of ) represent variables and variables The variance of Represents the correlation coefficient, which is used to reflect the variables and variables The linear correlation between Represents a constant.

[0034] According to a conditional preference optimization method for multimodal information processing provided by the present invention, the objective function of the self-attention reward strategy is:

[0035] ,

[0036] In the formula, represents the self-attention reward strategy loss, represents the sigmoid function used to convert scores into probabilities, Represents a multimodal large language model in the original question context image Generate the best answer The probability of Represents the preset reference strategy / prior distribution in the original problem-related image Generate the best answer The probability of represents a hyperparameter used to adjust the degree to which the multimodal large language model deviates from the reference model, represents an anchor value to ensure that the reward for the preferred response is always greater than the anchor value.

[0037] The present invention also provides a conditional preference optimization system for multimodal information processing, comprising:

[0038] The data acquisition module is used to: acquire user input data, wherein the user input data includes a question-related image and a question text;

[0039] A generation module is used to: obtain, based on user input data, a preference answer data pair corresponding to the image associated with the question and a preference answer data pair corresponding to the question text through a multimodal large language model;

[0040] The iterative optimization module is used to obtain the optimal answer corresponding to the user input data by using an improved conditional preference optimization algorithm based on the user input data and the preference answer data pairs corresponding to the question-associated image and the preference answer data pairs corresponding to the question text, wherein the improved conditional preference optimization algorithm includes a direct preference optimization algorithm, a multi-objective preference optimization algorithm and a self-attention reward strategy.

[0041] The present invention also provides an electronic device, comprising a processor and a memory storing a computer program, wherein the processor implements any of the above-mentioned conditional preference optimization methods for multimodal information processing when executing the computer program.

[0042] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described conditional preference optimization methods for multimodal information processing.

[0043] The present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute any of the conditional preference optimization methods for multimodal information processing described above.

[0044] The present invention provides a conditional preference optimization method, system, device and medium for multimodal information processing. The method constructs a preference answer data pair corresponding to a question-associated image and a preference answer data pair corresponding to a question text, optimizes the preference answer data pair corresponding to the question text using a direct preference optimization algorithm, and optimizes the preference answer data pair corresponding to the question-associated image using a multi-objective preference optimization algorithm. At the same time, a self-attention reward strategy is introduced to ensure that during the optimization process, even if the model learns a relative preference, the reward (or likelihood) of the preferred response will not decrease, thereby maintaining the quality of the preferred response and improving the multimodal large language model in processing multimodal inputs such as images and texts. The method can more accurately reflect human preferences, while avoiding the problem of relying solely on language information and ignoring image information, especially in reducing hallucination generation, and reducing the risk of decreased quality of the preferred response during the optimization process. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1One of the flow charts of a conditional preference optimization method for multimodal information processing provided by the present invention.

[0047] Figure 2 The second flowchart of the conditional preference optimization method for multimodal information processing provided by the present invention.

[0048] Figure 3 The third flowchart of the conditional preference optimization method for multimodal information processing provided by the present invention.

[0049] Figure 4 The present invention is a flowchart of an embodiment of a conditional preference optimization method for multimodal information processing provided by the present invention.

[0050] Figure 5 A schematic diagram of the structure of a conditional preference optimization system for multimodal information processing provided by the present invention.

[0051] Figure 6 This is a schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments, and they should not be understood as limitations on the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In the description of the present invention, it should be understood that the terms used are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0053] Combine the following Figure 1-Figure 6 The present invention describes the conditional preference optimization method, system, device and medium for multimodal information processing.

[0054] Reference Figure 1-Figure 3 The present invention provides a conditional preference optimization method for multimodal information processing, which may include:

[0055] Step S110, obtaining user input data, wherein the user input data includes a question-related image and a question text;

[0056] Step S120: obtaining, based on the user input data, a preference answer data pair corresponding to the question-associated image and a preference answer data pair corresponding to the question text through a multimodal large language model;

[0057] Step S130, based on the user input data and the preference answer data pairs corresponding to the question-associated image and the preference answer data pairs corresponding to the question text, use the improved conditional preference optimization algorithm to obtain the optimal answer corresponding to the user input data, wherein the improved conditional preference optimization algorithm includes a direct preference optimization algorithm, a multi-objective preference optimization algorithm and a self-attention reward strategy.

[0058] It should be noted that the executor of the conditional preference optimization method for multimodal information processing provided by the present invention can be any network-side device that meets the technical requirements, such as a conditional preference optimization device for multimodal information processing.

[0059] The conditional preference optimization method for multimodal information processing provided by the present invention can be applied to different fields, such as popular science (users input question-related images and question texts, and the present invention obtains popular science information, such as animal and plant science), inspection (users input question-related images and question texts, and the present invention identifies abnormal situations in the images and links to the knowledge base to provide emergency plans), etc. In one embodiment, users can input any question-related images and question texts through devices in the fields of virtual assistants, content creation, and intelligent recommendation systems. For example, see Figure 4 In the example shown, the question text that the user may enter is "What are the possible behaviors of the birds in the picture?", and the input question-associated image is a bird on a lawn.

[0060] In one embodiment, step S120 may include:

[0061] According to the user input data, a target image related to the question text is obtained from the question-related image through a multimodal large language model, and the answer data corresponding to the target image is obtained, forming a preference answer data pair with a one-to-one relationship with the target image as the preference answer data pair corresponding to the question-related image, wherein the target image can be multiple, and a preference answer data pair with a one-to-one relationship with each target image will be obtained, so that the iterative optimization result is better;

[0062] According to the user input data, the answer data corresponding to the question text is obtained through a multimodal large language model, and a preference answer data pair having a one-to-one relationship with the question text is formed as the preference answer data pair corresponding to the question text.

[0063] Specifically, step S120 can obtain a target image related to the question text from the question-associated image based on the user input data through a multimodal large language model and a target detection algorithm; or, according to the user input data, obtain a target image related to the question text from the question-associated image through a multimodal large language model, wherein the multimodal large language model generates a large model for image description.

[0064] In one embodiment, step S130 may include:

[0065] The preference answer data pairs corresponding to the question text are optimized using the direct preference optimization algorithm, and a direct preference optimization loss is obtained in each round of optimization.

[0066] According to the user input data and the preference answer data pairs corresponding to the image associated with the question, the preference answer data pairs corresponding to the image associated with the question are optimized using a multi-objective preference optimization algorithm, and a multi-objective preference optimization loss is obtained in each round of optimization process;

[0067] Using the self-attention reward strategy, a reward anchor is introduced to ensure that the reward of the preferred response remains positive during the optimization process, and a self-attention reward strategy loss is obtained in each round of optimization;

[0068] According to the above steps, multiple rounds of iterative optimization are performed. In each round of optimization, the sum of the direct preference optimization loss, multi-objective preference optimization loss, and self-attention reward strategy loss of that round is used as the total loss of that round, until the total loss meets the preset range, that is, the answer data corresponding to the total loss is used as the optimal answer corresponding to the user input data. Among them, the setting of the total loss to meet the preset range in this embodiment is intended to minimize the total loss within an acceptable range of optimization so as to ensure the optimization accuracy.

[0069] Among them, the overall objective function of the improved conditional preference optimization algorithm is:

[0070] ,

[0071] In the formula, represents the sum of direct preference optimization loss, multi-objective preference optimization loss, and self-attention reward strategy loss, denotes a direct preference for optimizing loss, represents the multi-objective preference optimization loss, Represents the self-attention reward strategy loss.

[0072] According to a conditional preference optimization method for multimodal information processing provided by the present invention, the objective function of the direct preference optimization algorithm is:

[0073] ,

[0074] In the formula, denotes a direct preference for optimizing loss, represents the sigmoid function used to convert scores into probabilities, Represents the question text of the multimodal large language model in the user input data Generate the best answer The probability of Represents the question text of the multimodal large language model in the user input data Generate non-preferred answers The probability of Represents the problem text of the preset reference strategy / prior distribution over the user input data Generate the best answer The probability of Represents the problem text of the preset reference strategy / prior distribution over the user input data Generate non-preferred answers The probability of represents a hyperparameter used to adjust the degree to which the multimodal large language model deviates from the reference model, represents an anchor value to ensure that the reward for the preferred response is always greater than the anchor value.

[0075] According to a conditional preference optimization method for multimodal information processing provided by the present invention, the objective function of the multi-objective preference optimization algorithm is:

[0076] ,

[0077] In the formula, represents the multi-objective preference optimization loss, represents the sigmoid function used to convert scores into probabilities, Represents a multimodal large language model in the question text and the target image Generate the target image below The probability of Represents a multimodal large language model in the question text Associate image with the original question Generate the original image below The probability of Indicates the preset reference strategy / prior distribution in the question text and the target image Generate the target image below and the target image is not the preferred answer The probability of Indicates the preset reference strategy / prior distribution in the question text Associate image with the original question Generate the original image below and the original image non-preferred answer The probability of represents a hyperparameter used to adjust the degree to which the multimodal large language model deviates from the reference model, represents an anchor value to ensure that the reward for the preferred response is always greater than the anchor value.

[0078] According to a conditional preference optimization method for multimodal information processing provided by the present invention, the joint probability density function of the preset reference strategy / prior distribution is:

[0079] ,

[0080] In the formula, ( ) represent variables and variables The mean vector of ) represent variables and variables The variance of Represents the correlation coefficient, which is used to reflect the variables and variables The degree of linear correlation between .

[0081] According to a conditional preference optimization method for multimodal information processing provided by the present invention, the objective function of the self-attention reward strategy is:

[0082] ,

[0083] In the formula, represents the self-attention reward strategy loss, represents the sigmoid function used to convert scores into probabilities, Represents a multimodal large language model in the original question context image Generate the best answer The probability of Represents the preset reference strategy / prior distribution in the original problem-related image Generate the best answer The probability of represents the hyperparameter used to adjust the optimization intensity, represents an anchor value to ensure that the reward for the preferred response is always greater than the anchor value.

[0084] In view of the problems existing in the prior art, the present invention proposes a conditional preference optimization method for multimodal information processing, which receives a user's question text and a question-related image as input data, constructs a preference answer data pair after being processed by a multimodal large language model, wherein a preference data pair acquisition strategy is designed, and the process is to acquire and intercept a target image related to the user's question from an original image with the help of the target recognition capability of the multimodal large language model, and then acquire text description information of the target image to form an image and answer preference data pair. For each pair of preference data, it usually includes a context (such as a question text and a question-related image), and two or more possible responses; at the same time, with the help of the language knowledge retrieval and generation capability of the multimodal large language model, the knowledge blocks related to the user's question are retrieved and acquired to form a language preference data pair; the language preference data pair obtains the reward difference between the preferred answer and the rejected answer through a direct preference optimization algorithm, and maximizes this difference, and derives the preference by comparing the relative probabilities of the two answers. The image and answer preference data are obtained through a multi-objective conditional preference optimization algorithm to obtain the preference differences of each intercepted target image and text description information. The preference differences of the original image, target image, image description, and retrieved knowledge information are introduced through a self-attention reward mechanism to finally generate a high-quality answer that best matches the image and question text associated with the user's input question. For a specific example process, please refer to Figure 4 .

[0085] The present invention provides a conditional preference optimization method, system, device and medium for multimodal information processing, which aims to overcome the shortcomings of the prior art in multimodal preference learning. By introducing conditional preference optimization and self-attention reward strategy, it is ensured that the model not only takes into account language preference during the optimization process, but also fully considers the visual information conditions of non-language modalities such as images, thereby avoiding the unconditional preference problem and effectively improving the model performance, especially in reducing the generation of hallucinations.

[0086] The present invention provides a conditional preference optimization method, system, device and medium for multimodal information processing. The method constructs a preference answer data pair corresponding to a question-associated image and a preference answer data pair corresponding to a question text, optimizes the preference answer data pair corresponding to the question text using a direct preference optimization algorithm, and optimizes the preference answer data pair corresponding to the question-associated image using a multi-objective preference optimization algorithm. At the same time, a self-attention reward strategy is introduced to ensure that during the optimization process, even if the model learns a relative preference, the reward (or likelihood) of the preferred response will not decrease, thereby maintaining the quality of the preferred response and improving the multimodal large language model in processing multimodal inputs such as images and texts. The method can more accurately reflect human preferences, avoid relying solely on language information and ignoring image information, and reduce the risk of a decrease in the quality of the preferred response during the optimization process.

[0087] The conditional preference optimization system for multimodal information processing provided by the present invention is described below. The conditional preference optimization system for multimodal information processing described below and the conditional preference optimization method for multimodal information processing described above can be referenced to each other.

[0088] Reference Figure 5 The present invention provides a conditional preference optimization system for multimodal information processing, which may include:

[0089] The data acquisition module is used to: acquire user input data, wherein the user input data includes a question-related image and a question text;

[0090] A generation module is used to: obtain, based on user input data, a preference answer data pair corresponding to the image associated with the question and a preference answer data pair corresponding to the question text through a multimodal large language model;

[0091] The iterative optimization module is used to obtain the optimal answer corresponding to the user input data by using an improved conditional preference optimization algorithm based on the user input data and the preference answer data pairs corresponding to the question-associated image and the preference answer data pairs corresponding to the question text, wherein the improved conditional preference optimization algorithm includes a direct preference optimization algorithm, a multi-objective preference optimization algorithm and a self-attention reward strategy.

[0092] In one embodiment, the generation module may include:

[0093] The first generation submodule is used to: obtain a target image related to the question text from the question-related image through a multimodal large language model according to the user input data, and obtain answer data corresponding to the target image, so as to form a preference answer data pair having a one-to-one relationship with the target image as the preference answer data pair corresponding to the question-related image;

[0094] The second generation submodule is used to: obtain answer data corresponding to the question text based on user input data through a multimodal large language model, and form a preference answer data pair with a one-to-one relationship with the question text as the preference answer data pair corresponding to the question text.

[0095] In one embodiment, the iterative optimization module may include:

[0096] The direct preference optimization submodule is used to: optimize the preference answer data pairs corresponding to the question text using the direct preference optimization algorithm, and obtain a direct preference optimization loss in each round of optimization;

[0097] The multi-objective preference optimization submodule is used to optimize the preference answer data pairs corresponding to the image associated with the question using a multi-objective preference optimization algorithm according to the user input data and the preference answer data pairs corresponding to the image associated with the question, and obtain a multi-objective preference optimization loss in each round of optimization process;

[0098] The self-attention reward submodule is used to: use the self-attention reward strategy, introduce a reward anchor point to ensure that the reward of the preferred response remains positive during the optimization process, and obtain a self-attention reward strategy loss in each round of optimization;

[0099] The iterative submodule is used to perform multiple rounds of iterative optimization according to the above steps. In each round of optimization, the sum of the direct preference optimization loss, multi-objective preference optimization loss, and self-attention reward strategy loss of that round is used as the total loss of that round, until the total loss meets the preset range, that is, the answer data corresponding to the total loss is used as the optimal answer corresponding to the user input data.

[0100] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the conditional preference optimization method for multimodal information processing, and the method includes:

[0101] Acquire user input data, wherein the user input data includes a question-related image and a question text;

[0102] According to the user input data, a preference answer data pair corresponding to the question-related image and a preference answer data pair corresponding to the question text are obtained through a multimodal large language model;

[0103] According to the user input data and the preference answer data pairs corresponding to the question-associated image and the preference answer data pairs corresponding to the question text, an improved conditional preference optimization algorithm is used to obtain the optimal answer corresponding to the user input data, wherein the improved conditional preference optimization algorithm includes a direct preference optimization algorithm, a multi-objective preference optimization algorithm and a self-attention reward strategy.

[0104] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0105] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the conditional preference optimization method for multimodal information processing provided by the above methods, the method comprising:

[0106] Acquire user input data, wherein the user input data includes a question-related image and a question text;

[0107] According to the user input data, a preference answer data pair corresponding to the question-related image and a preference answer data pair corresponding to the question text are obtained through a multimodal large language model;

[0108] According to the user input data and the preference answer data pairs corresponding to the question-associated image and the preference answer data pairs corresponding to the question text, an improved conditional preference optimization algorithm is used to obtain the optimal answer corresponding to the user input data, wherein the improved conditional preference optimization algorithm includes a direct preference optimization algorithm, a multi-objective preference optimization algorithm and a self-attention reward strategy.

[0109] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the conditional preference optimization method for multimodal information processing provided by the above methods, the method comprising:

[0110] Acquire user input data, wherein the user input data includes a question-related image and a question text;

[0111] According to the user input data, a preference answer data pair corresponding to the question-related image and a preference answer data pair corresponding to the question text are obtained through a multimodal large language model;

[0112] According to the user input data and the preference answer data pairs corresponding to the question-associated image and the preference answer data pairs corresponding to the question text, an improved conditional preference optimization algorithm is used to obtain the optimal answer corresponding to the user input data, wherein the improved conditional preference optimization algorithm includes a direct preference optimization algorithm, a multi-objective preference optimization algorithm and a self-attention reward strategy.

[0113] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0114] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A conditional preference optimization method for multimodal information processing, characterized in that: include: Acquire user input data, wherein the user input data includes a question-related image and a question text; According to the user input data, a preference answer data pair corresponding to the question-related image and a preference answer data pair corresponding to the question text are obtained through a multimodal large language model; According to the user input data and the preference answer data pairs corresponding to the image associated with the question and the preference answer data pairs corresponding to the question text, an improved conditional preference optimization algorithm is used to obtain the optimal answer corresponding to the user input data, wherein the improved conditional preference optimization algorithm includes a direct preference optimization algorithm, a multi-objective preference optimization algorithm and a self-attention reward strategy; According to the user input data and the preference answer data pair corresponding to the question-associated image and the preference answer data pair corresponding to the question text, an improved conditional preference optimization algorithm is used to obtain the optimal answer corresponding to the user input data, wherein the improved conditional preference optimization algorithm includes a direct preference optimization algorithm, a multi-objective preference optimization algorithm and a self-attention reward strategy, including: The preference answer data pairs corresponding to the question text are optimized using the direct preference optimization algorithm, and a direct preference optimization loss is obtained in each round of optimization. According to the user input data and the preference answer data pairs corresponding to the image associated with the question, the preference answer data pairs corresponding to the image associated with the question are optimized using a multi-objective preference optimization algorithm, and a multi-objective preference optimization loss is obtained in each round of optimization process; Using the self-attention reward strategy, a reward anchor is introduced to ensure that the reward of the preferred response remains positive during the optimization process, and a self-attention reward strategy loss is obtained in each round of optimization; Perform multiple rounds of iterative optimization according to the above steps. In each round of optimization, the sum of the direct preference optimization loss, multi-objective preference optimization loss, and self-attention reward strategy loss of that round is used as the total loss of that round until the total loss meets the preset range. That is, the answer data corresponding to the total loss is used as the optimal answer corresponding to the user input data. Among them, the objective function of the multi-objective preference optimization algorithm is: , In the formula, represents the multi-objective preference optimization loss, represents the sigmoid function used to convert scores into probabilities, Represents a multimodal large language model in the question text and the target image Generate the target image below The probability of Represents a multimodal large language model in the question text Associate image with the original question Generate the original image below The probability of Indicates the preset reference strategy / prior distribution in the question text and the target image Generate the target image below and the target image is not the preferred answer The probability of Indicates the preset reference strategy / prior distribution in the question text Associate image with the original question Generate the original image below and the original image non-preferred answer The probability of represents a hyperparameter used to adjust the degree to which the multimodal large language model deviates from the reference model, Represent an anchor value to ensure that the reward for the preferred response is always greater than the anchor value; Among them, the objective function of the self-attention reward strategy is: , In the formula, represents the self-attention reward strategy loss, represents the sigmoid function used to convert scores into probabilities, Represents a multimodal large language model in the original question context image Generate the best answer The probability of Represents the preset reference strategy / prior distribution in the original problem-related image Generate the best answer The probability of represents the hyperparameter used to adjust the optimization intensity, represents an anchor value to ensure that the reward for the preferred response is always greater than the anchor value.

2. The conditional preference optimization method for multimodal information processing according to claim 1, characterized in that: The method of obtaining, based on the user input data, a preference answer data pair corresponding to the question-associated image and a preference answer data pair corresponding to the question text through a multimodal large language model includes: According to the user input data, a target image related to the question text is obtained from the question-related image through a multimodal large language model, and answer data corresponding to the target image is obtained, so as to form a preference answer data pair having a one-to-one relationship with the target image as the preference answer data pair corresponding to the question-related image; According to the user input data, the answer data corresponding to the question text is obtained through a multimodal large language model, and a preference answer data pair having a one-to-one relationship with the question text is formed as the preference answer data pair corresponding to the question text.

3. The conditional preference optimization method for multimodal information processing according to claim 1, characterized in that: The overall objective function of the improved conditional preference optimization algorithm is: , In the formula, represents the sum of direct preference optimization loss, multi-objective preference optimization loss, and self-attention reward strategy loss, denotes a direct preference for optimizing loss, represents the multi-objective preference optimization loss, Represents the self-attention reward strategy loss.

4. The conditional preference optimization method for multimodal information processing according to claim 3, characterized in that: The objective function of the direct preference optimization algorithm is: , In the formula, denotes a direct preference for optimizing loss, represents the sigmoid function used to convert scores into probabilities, Represents the question text of the multimodal large language model in the user input data Generate the best answer The probability of Represents the question text of the multimodal large language model in the user input data Generate non-preferred answers The probability of Represents the problem text of the preset reference strategy / prior distribution over the user input data Generate the best answer The probability of Represents the problem text of the preset reference strategy / prior distribution over the user input data Generate non-preferred answers The probability of represents a hyperparameter used to adjust the degree to which the multimodal large language model deviates from the reference model, represents an anchor value to ensure that the reward for the preferred response is always greater than the anchor value.

5. A conditional preference optimization system for multimodal information processing, characterized in that: include: The data acquisition module is used to: acquire user input data, wherein the user input data includes a question-related image and a question text; A generation module is used to: obtain, based on user input data, a preference answer data pair corresponding to the image associated with the question and a preference answer data pair corresponding to the question text through a multimodal large language model; An iterative optimization module is used to obtain the optimal answer corresponding to the user input data by using an improved conditional preference optimization algorithm according to the user input data and the preference answer data pairs corresponding to the image associated with the question and the preference answer data pairs corresponding to the question text, wherein the improved conditional preference optimization algorithm includes a direct preference optimization algorithm, a multi-objective preference optimization algorithm and a self-attention reward strategy; Among them, the iterative optimization module includes: The direct preference optimization submodule is used to: optimize the preference answer data pairs corresponding to the question text using the direct preference optimization algorithm, and obtain a direct preference optimization loss in each round of optimization; The multi-objective preference optimization submodule is used to optimize the preference answer data pairs corresponding to the image associated with the question using a multi-objective preference optimization algorithm according to the user input data and the preference answer data pairs corresponding to the image associated with the question, and obtain a multi-objective preference optimization loss in each round of optimization process; The self-attention reward submodule is used to: use the self-attention reward strategy, introduce a reward anchor point to ensure that the reward of the preferred response remains positive during the optimization process, and obtain a self-attention reward strategy loss in each round of optimization; The iterative submodule is used to: perform multiple rounds of iterative optimization according to the above steps, and in each round of optimization, the sum of the direct preference optimization loss, the multi-objective preference optimization loss, and the self-attention reward strategy loss of the round is used as the total loss of the round, until the total loss meets the preset range, that is, the answer data corresponding to the total loss is used as the optimal answer corresponding to the user input data; Among them, the objective function of the multi-objective preference optimization algorithm is: , In the formula, represents the multi-objective preference optimization loss, represents the sigmoid function used to convert scores into probabilities, Represents a multimodal large language model in the question text and the target image Generate the target image below The probability of Represents a multimodal large language model in the question text Associate image with the original question Generate the original image below The probability of Indicates the preset reference strategy / prior distribution in the question text and the target image Generate the target image below and the target image is not the preferred answer The probability of Indicates the preset reference strategy / prior distribution in the question text Associate image with the original question Generate the original image below and the original image non-preferred answer The probability of represents a hyperparameter used to adjust the degree to which the multimodal large language model deviates from the reference model, Represent an anchor value to ensure that the reward for the preferred response is always greater than the anchor value; Among them, the objective function of the self-attention reward strategy is: , In the formula, represents the self-attention reward strategy loss, represents the sigmoid function used to convert scores into probabilities, Represents a multimodal large language model in the original question context image Generate the best answer The probability of Represents the preset reference strategy / prior distribution in the original problem-related image Generate the best answer The probability of represents the hyperparameter used to adjust the optimization intensity, represents an anchor value to ensure that the reward for the preferred response is always greater than the anchor value.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the conditional preference optimization method for multimodal information processing as described in any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the conditional preference optimization method for multimodal information processing as described in any one of claims 1 to 4 is implemented.

Citation Information

Cited By

  • Language model training for direct preference optimization and improved language model

    US12694301B1