A multi-modal preference optimization method and system based on bidirectional distribution alignment

Abstract

The present application belongs to the technical field of multi-modal large model alignment and preference learning, and particularly relates to a multi-modal preference optimization method and system based on bidirectional distribution alignment. The method generates candidate response sets corresponding to the original image and the edited image respectively based on the original image, the edited image and the question under the current model parameter distribution, then for each candidate response, positive samples are screened based on the confidence score and the reference consistency score with the reference response, and negative samples are screened through an entropy-guided negative sample mining strategy; subsequently, the positive and negative samples corresponding to the original image and the edited image are input into the model, the overall loss of the model is calculated based on the image-level and response-level contrastive loss, and the model parameter distribution is updated until the model converges. The present application significantly reduces the hallucination rate through bidirectional alignment of the model parameter distribution and the data distribution, and introduces an entropy-guided negative sample mining strategy in the direction of data adaptation model, thereby realizing targeted hallucination suppression.

Description

Technical Field

[0001] This invention belongs to the field of multimodal large model alignment and preference learning technology, specifically involving a multimodal preference optimization method and system based on bidirectional distribution alignment. Background Technology

[0002] With the development of large-scale visual language models, significant progress has been made in image understanding, visual question answering (VQA), and reasoning tasks. However, current models still commonly suffer from the multimodal hallucination problem, manifested as inconsistencies between generated content and visual evidence, or fictitious objects or attributes. These problems severely impact model credibility and practical deployment.

[0003] Among existing methods, reinforcement learning from human feedback (RLHF) based on reward modeling or human feedback suffers from high optimization costs and large sample requirements. Direct preference optimization (DPO) does not require explicit construction of a reward model and directly optimizes preference pairs, achieving stable convergence. However, it heavily relies on online policy data consistent with the current model. If the training data comes from offline policy data, it will lead to KL divergence, gradient instability, and alignment failure, thus affecting the reliability of the learning effect. On-Policy Alignment Direct Preference Optimization (OPA-DPO) alleviates the problems of traditional DPO through LoRA (low-rank adaptation) technology. The problem of poor performance under offline policy data is addressed by simply projecting the corrected response into the model policy space and correcting it in one direction (from model to data). This makes the optimized model passively adapt to the static data distribution, but ignores the adaptability of the data distribution to the model. It cannot achieve two-way dynamic collaboration between the model and the data, and it is difficult to proactively optimize the perception of data quality and uncertainty. Summary of the Invention

[0004] The purpose of this invention is to address the aforementioned problems in the prior art by providing a multimodal preference optimization method and system based on bidirectional distribution alignment, which improves multimodal consistency through a bidirectional alignment model and data distribution, thereby reducing the illusion problem.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] In a first aspect, the present invention provides a multimodal preference optimization method based on bidirectional distribution alignment, the multimodal preference optimization method comprising:

[0007] S1. Based on the input original image, edited image, and question, generate a candidate response set corresponding to the original image and a candidate response set corresponding to the edited image under the current model strategy;

[0008] S2. For each candidate response in the candidate response set, positive samples are selected from the candidate response set based on its confidence score and reference consistency score with the reference response, and negative samples are selected from the candidate response set through an entropy-guided negative sample mining strategy.

[0009] S3. Input the positive and negative samples selected from the candidate response set corresponding to the original image and the positive and negative samples selected from the candidate response set corresponding to the edited image into the model, and calculate the contrast loss in both the image level and the response level directions respectively.

[0010] S4. Calculate the overall model loss based on the contrast loss in both the image and response directions, and update the model strategy based on the overall model loss; repeat the above steps until the model converges.

[0011] In S2, when positive samples cannot be obtained from the candidate response set corresponding to the original image or the candidate response set corresponding to the edited image, the model strategy is dynamically reconstructed through an adaptive distribution shrinkage strategy. Under the reconstructed model strategy, the on-policy strategy is used to regenerate the candidate response set corresponding to the original image or the candidate response set corresponding to the edited image.

[0012] The adaptive distribution shrinkage strategy includes: firstly, calculating the sampling weight based on the mutual information gain of each candidate response to the reference response, the cross-candidate consistency among candidate responses in the candidate response set, and the length regularization term of each candidate response; and then obtaining the reconstructed model strategy based on the sampling weight.

[0013] The formula for calculating the mutual information gain of each candidate response with respect to the reference response is as follows:

[0014] ;

[0015] In the above formula, For the candidate response set Each candidate response is relative to the reference response. Mutual information gain; For the first The edge entropy of each candidate response; In a given reference response Next The conditional entropy of each candidate response;

[0016] The formula for calculating cross-candidate consistency among candidate responses in the candidate response set is as follows:

[0017] ;

[0018] In the above formula, To ensure cross-candidate consistency among candidate responses within the candidate response set; The total number of candidate responses in the candidate response set; This indicates that the first encoder will be embedded. One candidate response or the One reference response Mapped to a high-dimensional semantic vector;

[0019] The formula for calculating the length regularization term of each candidate response is as follows:

[0020] ;

[0021] In the above formula, For the first The length regularization term for each candidate response; The regularization coefficient is used. For the first The token length of each candidate response; The average token length of all candidate responses in the candidate response set.

[0022] The formula for calculating the sampling weight is:

[0023] ;

[0024] In the above formula, For the first The sampling weights of each candidate response; For weighted discrimination;

[0025] The reconstructed model strategy is obtained based on the sampling weights using the following formula:

[0026] ;

[0027] In the above formula, For the reconstructed model strategy Below, given image and the problem Generate response The probability of; In the current model policy Below, given image and the problem Generate response The probability of; , For the original image, For editing images; This is the shrinkage strength coefficient.

[0028] In S2, the entropy-guided negative sample mining strategy includes: for candidate responses ,in Given the token length, calculate the candidate response. The dynamic entropy threshold; for candidate responses The first in tokens Calculate the first tokens Predicted entropy; candidate responses Tokens whose predicted entropy exceeds the maximum threshold are marked as high-entropy fragments, while the remaining tokens are marked as low-entropy fragments. Then, the CLIP score of the obtained high-entropy and low-entropy fragments with the reference image is calculated for image-text consistency detection, and the BERT score of the fragments with the reference response is calculated for semantic consistency detection. If a high-entropy fragment or a low-entropy fragment fails to meet either of these detection criteria, it is determined as a negative sample.

[0029] The formula for calculating the maximum threshold is:

[0030] ;

[0031] In the above formula, Candidate response The dynamic entropy threshold; This is the mean of the predicted entropy of all tokens within the batch; The standard deviation of the predicted entropy for all tokens within the batch;

[0032] The first tokens The formula for calculating the prediction entropy is:

[0033] ;

[0034] In the above formula, Predicted entropy for each token; Indicates for the first tokens Previously, the model had generated predicted distributions for all historical token sequences; This indicates that all tokens are iterated over.

[0035] In S2, the step of selecting positive samples based on confidence scores and reference consistency scores includes: weighting the confidence scores and reference consistency scores to obtain an overall quality score, and determining whether the overall quality score exceeds a preset threshold. If so, then the candidate response is taken as a positive sample. .

[0036] Calculate the confidence score using the following formula:

[0037] ;

[0038] In the above formula, Represents a given image and the problem Candidate responses generated by model sampling; , For the original image, For editing images; Candidate response Confidence score; for The Middle One token; For the front A prefix consisting of tokens; Candidate response The sequence length;

[0039] The reference consistency score is calculated using the following formula:

[0040] ;

[0041] In the above formula, Candidate response Reference Response Reference consistency score; This indicates that candidate responses are selected by embedding the encoder. Or refer to the response Mapped to a high-dimensional semantic vector; This represents the function for calculating cosine similarity.

[0042] The formula for calculating the overall quality score is as follows:

[0043] ;

[0044] In the above formula, Candidate response The overall quality score; To control the relative weights of confidence scores and reference consistency scores, a fusion coefficient is used.

[0045] In S4, the formula for calculating the overall loss of the model is:

[0046] ;

[0047] ;

[0048] ;

[0049] ;

[0050] ;

[0051] In the above formula, This represents the overall loss of the model. These are the weights used to balance the contrast loss in both the image-level and response-level directions; It is the sigmoid function; This is the preference scaling factor; To constrain the same problem The model processes the original image as input. Generated positive samples Compared to editing images Image-level contrast loss with higher preference scores; To constrain the same edited image Under the given conditions, the positive samples generated by the model Compared to the original image It has a higher preference score; To constrain the same original image Under these conditions, the model applies to positive samples. The preference is higher than that of negative samples Response level contrast loss; To edit the same image Under the given conditions, the positive samples generated by the model relative to negative samples Response level contrast loss; , These are positive and negative samples, respectively, selected from the candidate response set corresponding to the edited image. , These are positive and negative samples obtained from the candidate response set corresponding to the original image, respectively. Indicates the current model policy Below, given the problem and editing images Generate positive samples The probability of; Indicates a fixed model strategy Below, given the problem and editing images Generate positive samples The probability of; In the current model policy Below, given the problem and the original image Generate positive samples The probability of; For a fixed model strategy Below, given the problem and the original image Generate positive samples The probability of; Indicates the current model policy Below, given the problem and editing images Generate positive samples The probability of; Indicates a fixed model strategy Below, given the problem and editing images Generate positive samples The probability of; Indicates the current model policy Below, given the problem and the original image Generate negative samples The probability of; Indicates a fixed model strategy Below, given the problem and the original image Generate negative samples The probability of; Indicates the current model policy Below, given the problem and editing images Generate positive samples The probability of; Indicates a fixed model strategy Below, given the problem and editing images Generate positive samples The probability of; Indicates the current model policy Below, given the problem and editing images Generate negative samples The probability of; Indicates a fixed model strategy Below, given the problem and editing images Generate negative samples The probability of.

[0052] Secondly, the present invention provides a multimodal preference optimization system based on bidirectional distribution alignment. The multimodal preference optimization system is based on the aforementioned multimodal preference optimization method, and the system includes:

[0053] The on-policy candidate generation module is used to take the original image, the edited image, and the question as input, and generate a candidate response set corresponding to the original image and a candidate response set corresponding to the edited image under the current model policy.

[0054] The positive and negative sample screening module is used to select positive samples from the candidate response set based on the confidence score and the reference consistency score with the reference response for each candidate response in the candidate response set, and to select negative samples from the candidate response set through an entropy-guided negative sample mining strategy.

[0055] The bidirectional preference optimization module is used to input positive and negative samples selected from the candidate response set corresponding to the original image and positive and negative samples selected from the candidate response set corresponding to the edited image into the model, and calculate the contrast loss in both the image level and the response level.

[0056] The model update module is used to calculate the overall model loss based on the contrast loss in both the image level and response level directions, and update the model strategy based on the overall model loss until the model converges.

[0057] The positive and negative sample selection module includes a positive sample selection module and an adaptive distribution shrinkage module. The positive sample selection module is used to select positive samples from the candidate response set for each candidate response by calculating its confidence score and reference consistency score with the reference response. The adaptive distribution shrinkage module is used to dynamically reconstruct the model strategy when positive samples cannot be selected from the candidate response set corresponding to the original image or the candidate response set corresponding to the edited image. Under the reconstructed model strategy, an on-policy strategy is used to regenerate the candidate response set corresponding to the original image or the candidate response set corresponding to the edited image. The adaptive distribution shrinkage strategy includes: firstly, calculating the sampling weight based on the mutual information gain of each candidate response to the reference response, the cross-candidate consistency among candidate responses in the candidate response set, and the length regularization term of each candidate response; and then obtaining the reconstructed model strategy based on the sampling weight.

[0058] The adaptive distribution shrinking module is used to calculate the mutual information gain of each candidate response with respect to the reference response according to the following formula:

[0059] ;

[0060] In the above formula, For the candidate response set Each candidate response is relative to the reference response. Mutual information gain; For the first The edge entropy of each candidate response; In a given reference response Next The conditional entropy of each candidate response;

[0061] The adaptive distribution shrinking module is used to calculate cross-candidate consistency among candidate responses in the candidate response set according to the following formula:

[0062] ;

[0063] In the above formula, To ensure cross-candidate consistency among candidate responses within the candidate response set; The total number of candidate responses in the candidate response set; This indicates that the first encoder will be embedded. One candidate response or the One reference response Mapped to a high-dimensional semantic vector;

[0064] The adaptive distribution shrinking module is used to calculate the length regularization term for each candidate response according to the following formula:

[0065] ;

[0066] In the above formula, For the first The length regularization term for each candidate response; The regularization coefficient is used. For the first The token length of each candidate response; The average token length of all candidate responses in the candidate response set.

[0067] The adaptive distribution shrinkage module is used to calculate the sampling weights according to the following formula:

[0068] ;

[0069] In the above formula, For the first The sampling weights of each candidate response; For weighted discrimination;

[0070] The adaptive distribution shrinkage module is used to obtain the reconstructed model strategy based on the sampling weights according to the following formula:

[0071] ;

[0072] In the above formula, For the reconstructed model strategy Below, given image and the problem Generate response The probability of; In the current model policy Below, given image and the problem Generate response The probability of; , For the original image, For editing images; This is the shrinkage strength coefficient.

[0073] The positive and negative sample screening module also includes an entropy-guided negative sample mining module, which is used for candidate responses. ,in Given the token length, calculate the candidate response. The dynamic entropy threshold; for candidate responses The first in tokens Calculate the first tokens Predicted entropy; candidate responses Tokens whose predicted entropy exceeds the maximum threshold are marked as high-entropy fragments, while the remaining tokens are marked as low-entropy fragments. Then, the CLIP score of the obtained high-entropy and low-entropy fragments with the reference image is calculated for image-text consistency detection, and the BERT score of the fragments with the reference response is calculated for semantic consistency detection. If a high-entropy fragment or a low-entropy fragment fails to meet either of these detection criteria, it is determined as a negative sample.

[0074] The entropy-guided negative sample mining module is used to calculate the maximum threshold according to the following formula:

[0075] ;

[0076] In the above formula, Candidate response The dynamic entropy threshold; This is the mean of the predicted entropy of all tokens within the batch; The standard deviation of the predicted entropy for all tokens within the batch;

[0077] The first tokens The formula for calculating the prediction entropy is:

[0078] ;

[0079] In the above formula, Predicted entropy for each token; Indicates for the first tokens Previously, the model had generated predicted distributions for all historical token sequences; This indicates that all tokens are iterated over.

[0080] The positive sample screening module is used to screen positive samples based on confidence score and reference consistency score according to the following steps: weighting the confidence score and reference consistency score to obtain the overall quality score, and determining whether the overall quality score exceeds a preset threshold. If so, then the candidate response is taken as a positive sample. .

[0081] The positive sample screening module is used to calculate the confidence score according to the following formula:

[0082] ;

[0083] In the above formula, Represents a given image and the problem Candidate responses generated by model sampling; , For the original image, For editing images; Candidate response Confidence score; for The Middle One token; For the front A prefix consisting of tokens; Candidate response The sequence length;

[0084] The positive sample screening module is used to calculate the reference consistency score according to the following formula:

[0085] ;

[0086] In the above formula, Candidate response Reference Response Reference consistency score; This indicates that candidate responses are selected by embedding the encoder. Or refer to the response Mapped to a high-dimensional semantic vector; This represents the function for calculating cosine similarity.

[0087] The positive sample screening module is used to calculate the overall quality score according to the following formula:

[0088] ;

[0089] In the above formula, Candidate response The overall quality score; To control the relative weights of confidence scores and reference consistency scores, a fusion coefficient is used.

[0090] The bidirectional preference optimization module is used to calculate the contrast loss in both the image level and response level directions according to the following formula:

[0091] ;

[0092] ;

[0093] ;

[0094] ;

[0095] In the above formula, It is the sigmoid function; This is the preference scaling factor; To constrain the same problem The model processes the original image as input. Generated positive samples Compared to editing images Image-level contrast loss with higher preference scores; To constrain the same edited image Under the given conditions, the positive samples generated by the model Compared to the original image It has a higher preference score; To constrain the same original image Under these conditions, the model applies to positive samples. The preference is higher than that of negative samples Response level contrast loss; To edit the same image Under the given conditions, the positive samples generated by the model relative to negative samples Response level contrast loss; , These are positive and negative samples, respectively, selected from the candidate response set corresponding to the edited image. , These are positive and negative samples obtained from the candidate response set corresponding to the original image, respectively. Indicates the current model policy Below, given the problem and editing images Generate positive samples The probability of; Indicates a fixed model strategy Below, given the problem and editing images Generate positive samples The probability of; In the current model policy Below, given the problem and the original image Generate positive samples The probability of; For a fixed model strategy Below, given the problem and the original image Generate positive samples The probability of; Indicates the current model policy Below, given the problem and editing images Generate positive samples The probability of; Indicates a fixed model strategy Below, given the problem and editing images Generate positive samples The probability of; Indicates the current model policy Below, given the problem and the original image Generate negative samples The probability of; Indicates a fixed model strategy Below, given the problem and the original image Generate negative samples The probability of; Indicates the current model policy Below, given the problem and editing images Generate positive samples The probability of; Indicates a fixed model strategy Below, given the problem and editing images Generate positive samples The probability of; Indicates the current model policy Below, given the problem and editing images Generate negative samples The probability of; Indicates a fixed model strategy Below, given the problem and editing images Generate negative samples The probability of;

[0096] The model update module is used to calculate the overall model loss according to the following formula:

[0097] ;

[0098] In the above formula, This represents the overall loss of the model. These are the weights used to balance the contrast loss in both the image-level and response-level directions.

[0099] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0100] 1. The multimodal preference optimization method based on bidirectional distribution alignment described in this invention proposes a bidirectional preference optimization framework applicable to large-scale visual language models. Through bidirectional alignment of model strategy and data distribution, it achieves mutual adaptation between model distribution and data distribution, thereby achieving stable and robust multimodal preference learning. Specifically, the model-to-data direction and the data-to-model direction together form a closed-loop adaptive alignment process. Through iterative optimization, the model and data are continuously refined, ultimately achieving distribution balance and significantly reducing the illusion rate. Compared with unidirectional model fitting of static preference data, the preference data distribution in the above design should be able to dynamically adapt to the evolutionary representation of the model, forming a co-evolutionary process between the model and data, which can achieve more stable and more generalizable multimodal alignment.

[0101] 2. The multimodal preference optimization method based on bidirectional distribution alignment described in this invention introduces adaptive distribution contraction in the model-to-data direction. That is, by weighting candidate responses through mutual information gain, cross-candidate consistency, and length regularization terms, the generated distribution is adaptively contracted to a reliable subspace, ensuring that the model maintains online policy stability while focusing on high-quality data. In the data-to-model direction, entropy-guided hard sample mining is introduced. That is, by estimating token-level entropy values ​​and combining them with multimodal alignment indicators such as CLIP scores and BERT scores, samples prone to hallucinations are identified. These high-uncertainty samples are reweighted as negative samples, guiding the model to focus on correcting hallucination-sensitive regions, thereby achieving targeted hallucination suppression. Attached Figure Description

[0102] Figure 1 This is a flowchart of the method described in this invention.

[0103] Figure 2 This is a schematic diagram of the bidirectional preference optimization framework constructed by the method described in this invention.

[0104] Figure 3 This is a comparison chart of the description results of the same image by the method described in this invention, LLaVA-1.5, traditional DPO, and OPA-DPO.

[0105] Figure 4 This is a structural block diagram of the system described in this invention. Detailed Implementation

[0106] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings.

[0107] Example 1:

[0108] See Figure 1 A multimodal preference optimization method based on bidirectional distribution alignment is performed in the following steps:

[0109] S1. Based on the input original image, edited image, and question, generate a candidate response set corresponding to the original image and a candidate response set corresponding to the edited image under the current model strategy.

[0110] Specifically, to avoid the off-policy offset problem in existing DPO-type methods, in the current model policy A candidate response set is generated, ensuring that all samples come from an on-policy distribution. Given an image... With the question From the current model strategy Medium sampling Each response generates a candidate response set, and the candidate responses in the candidate response set... Follows the following distribution: , , Represents the original image. This indicates that the image is being edited. For the original image... The corresponding candidate response set is represented as For editing images The corresponding candidate response set is represented as , This represents the total number of candidate responses in the candidate response set.

[0111] S2, for each candidate response in the candidate response set Positive samples are selected from the candidate response set based on the confidence score and the reference consistency score with the reference response, and negative samples are selected from the candidate response set through an entropy-guided negative sample mining strategy.

[0112] Specifically, the confidence score is calculated using the following formula:

[0113] ;

[0114] In the above formula, Represents a given image and the problem Candidate responses generated by model sampling; , For the original image, For editing images; Candidate response Confidence score; for The Middle One token; For the front A prefix consisting of tokens; Candidate response The sequence length;

[0115] The reference consistency score is calculated using the following formula:

[0116] ;

[0117] In the above formula, Candidate response Reference Response Reference consistency score; This indicates that candidate responses are selected by embedding the encoder. Or refer to the response Mapped to a high-dimensional semantic vector; This represents the function for calculating cosine similarity.

[0118] Specifically, the step of selecting positive samples based on confidence scores and reference consistency scores includes: calculating a weighted average of the confidence scores and reference consistency scores to obtain an overall quality score, and determining whether the overall quality score exceeds a preset threshold. If so, then the candidate response is taken as a positive sample. ,when The positive samples obtained at that time are denoted as ,when The positive samples obtained at that time are denoted as The preset threshold It can be set according to the statistical distribution of the quality score of candidate responses during the training or running phase, such as based on the mean, quantiles or dynamic statistics of historical samples, to distinguish between high-quality and low-quality responses.

[0119] The formula for calculating the overall quality score is as follows:

[0120] ;

[0121] In the above formula, Candidate response The overall quality score; To control the relative weights of confidence scores and reference consistency scores, a fusion coefficient is used.

[0122] Specifically, when no positive samples can be obtained from the candidate response set corresponding to the original image or the candidate response set corresponding to the edited image, the model strategy is dynamically reconstructed using an adaptive distribution shrinkage (ADC) strategy. Under the reconstructed model strategy, an on-policy strategy is used to regenerate the candidate response set corresponding to the original image or the candidate response set corresponding to the edited image. If no qualified candidate response can be selected as a positive sample from the regenerated candidate response set, the current training sample is skipped to avoid off-policy noise pollution. This invention, by dynamically reconstructing the model strategy, can shrink the model strategy to a subspace of high-quality responses.

[0123] Specifically, the adaptive distribution shrinkage strategy includes:

[0124] First, calculate the mutual information gain of each candidate response with respect to the reference response, the cross-candidate consistency among candidate responses in the candidate response set, and the length regularization term for each candidate response; the mutual information gain of each candidate response with respect to the reference response is used to measure the... One candidate response Relative to reference response The newly provided information; the formula for calculating mutual information gain is:

[0125] ;

[0126] In the above formula, For the candidate response set Each candidate response is relative to the reference response. Mutual information gain; For the first The edge entropy of each candidate response; In a given reference response Next The conditional entropy of each candidate response;

[0127] The cross-candidate consistency among candidate responses in the candidate response set is used to measure the stability of semantic features among candidates; the formula for calculating cross-candidate consistency is:

[0128] ;

[0129] In the above formula, To ensure cross-candidate consistency among candidate responses within the candidate response set; The total number of candidate responses in the candidate response set; This indicates that the first encoder will be embedded. One candidate response or the One reference response Mapped to a high-dimensional semantic vector;

[0130] The length regularization term for each candidate response is used to penalize responses that are significantly too long or too short; the formula for calculating the length regularization term is:

[0131] ;

[0132] In the above formula, For the first The length regularization term for each candidate response; The regularization coefficient is used. For the first The token length of each candidate response; The average token length of all candidate responses in the candidate response set;

[0133] Then, the sampling weight is calculated based on the mutual information gain of each candidate response to the reference response, the cross-candidate consistency among candidate responses in the candidate response set, and the length regularization term of each candidate response; the formula for calculating the sampling weight is as follows:

[0134] ;

[0135] In the above formula, For the first The sampling weights of each candidate response; The weight discrimination index is as follows: the larger the value, the more significant the difference in weight between candidates, and the more concentrated the sampling distribution is in the high-quality region; the smaller the value, the more gradual the difference in weight between candidates, and the ability to explore more candidate responses is preserved.

[0136] Then, based on the sampling weights, the reconstructed model strategy is obtained using the following formula:

[0137] ;

[0138] In the above formula, For the reconstructed model strategy Below, given image and the problem Generate response The probability of; In the current model policy Below, given Generate response The probability of; , For the original image, For editing images; This is the shrinkage strength coefficient.

[0139] Specifically, to construct effective negative samples, this invention introduces an entropy-guided negative sample mining strategy (EHSD) to analyze the uncertainty and text-image consistency of candidate responses at the token granularity, thereby constructing stable and information-rich difficult negative samples. The entropy-guided negative sample mining strategy includes: for candidate responses... ,in Given the token length, calculate the candidate response. The dynamic entropy threshold; for candidate responses The first in tokens Calculate the first tokens Predicted entropy; candidate responses Tokens whose predicted entropy exceeds the maximum threshold are marked as high-entropy fragments, while the remaining tokens are marked as low-entropy fragments. Then, the CLIP score of both high-entropy and low-entropy fragments is calculated with respect to the reference image for image-text consistency detection, and the BERT score is calculated with respect to the reference response for semantic consistency detection. If a high-entropy fragment fails to meet either of these detection criteria, it is determined to have weak illusion; if a low-entropy fragment fails to meet either of these detection criteria, it is determined to have strong illusion (confidence but error). Both categories are then classified as negative samples. The negative samples obtained at that time are denoted as ,when The negative samples obtained at that time are denoted as .

[0140] The formula for calculating the maximum threshold is:

[0141] ;

[0142] In the above formula, Candidate response The dynamic entropy threshold; This is the mean of the predicted entropy of all tokens within the batch; The standard deviation of the predicted entropy for all tokens within the batch;

[0143] The first tokens The formula for calculating the prediction entropy is:

[0144] ;

[0145] In the above formula, Predicted entropy for each token; Indicates for the first tokens Previously, the model had generated predicted distributions for all historical token sequences; This indicates that all tokens are iterated over.

[0146] S3. Positive and negative samples selected from the candidate response set corresponding to the original image and positive and negative samples selected from the candidate response set corresponding to the edited image are input into the model. Contrast loss is calculated at both the image level and the response level. The image-level contrast loss requires that positive samples should obtain higher preference scores on their matching images, and negative samples should obtain higher preference scores on their matching images, to ensure that the model has robust visual condition sensitivity between the original image and the edited image. The response-level contrast loss further strengthens the above preference relationship to ensure that the model prioritizes generating high-quality, low-illusion responses under the same visual conditions.

[0147] Specifically, the expressions for the contrast loss in both the image-level and response-level directions are as follows:

[0148] ;

[0149] ;

[0150] ;

[0151] ;

[0152] In the above formula, It is the sigmoid function; This is the preference scaling factor; To constrain the same problem The model processes the original image as input. Generated positive samples Compared to editing images Image-level contrast loss with higher preference scores; To constrain the same edited image Under the given conditions, the positive samples generated by the model Compared to the original image It has a higher preference score; To constrain the same original image Under these conditions, the model applies to positive samples. The preference is higher than that of negative samples Response level contrast loss; To edit the same image Under the given conditions, the positive samples generated by the model relative to negative samples Response level contrast loss; , These are positive and negative samples, respectively, selected from the candidate response set corresponding to the edited image. , These are positive and negative samples obtained from the candidate response set corresponding to the original image, respectively. Indicates the current model policy Below, given the problem and editing images Generate positive samples The probability of; Indicates a fixed model strategy Below, given the problem and editing images Generate positive samples The probability of; In the current model policy Below, given the problem and the original image Generate positive samples The probability of; For a fixed model strategy Below, given the problem and the original image Generate positive samples The probability of; Indicates the current model policy Below, given the problem and editing images Generate positive samples The probability of; Indicates a fixed model strategy Below, given the problem and editing images Generate positive samples The probability of; Indicates the current model policy Below, given the problem and the original image Generate negative samples The probability of; Indicates a fixed model strategy Below, given the problem and the original image Generate negative samples The probability of; Indicates the current model policy Below, given the problem and editing images Generate positive samples The probability of; Indicates a fixed model strategy Below, given the problem and editing images Generate positive samples The probability of; Indicates the current model policy Below, given the problem and editing images Generate negative samples The probability of; Indicates a fixed model strategy Below, given the problem and editing images Generate negative samples The probability of. and Both are conditional probability distribution models in bidirectional preference optimization methods, representing the current model policy to be optimized and a fixed reference model policy, respectively. The optimization is achieved by measuring the current model policy to be optimized. With fixed reference model strategy The differences between the two factors prevent the model from deviating from its original capabilities or exhibiting extreme behavior during the optimization process; together, they form the basis of preference optimization.

[0153] S4. Calculate the overall model loss based on the contrast loss in both the image and response directions, and finally achieve bidirectional alignment of the large visual language model in terms of model policy and preference data distribution; update the model policy based on the overall model loss; repeat the above steps until the model converges.

[0154] Specifically, the formula for calculating the overall loss of the model is as follows:

[0155] ;

[0156] In the above formula, This represents the overall loss of the model. These are the weights used to balance the contrast loss in both the image-level and response-level directions.

[0157] In traditional Data Processing (DPO), the model strategy is continuously updated to a fixed data distribution, but the data itself cannot perceive the model's evolutionary behavior. When the data exceeds the model's cognitive range, this imbalance leads to optimization divergence, causing the alignment effect to fail. To solve the above problems, the method described in this invention proposes... Figure 2The bidirectional preference optimization framework shown is suitable for large-scale visual language models and includes the following four modules: on-policy candidate generation based on model policy, adaptive distribution shrinkage, entropy-guided hard sample mining, and bidirectional preference comparison optimization. Adaptive distribution shrinkage can improve the stability of positive samples, entropy-guided hard sample mining can automatically discover high-risk samples of illusion, and bidirectional preference comparison optimization can significantly reduce off-policy bias. The above modules work together to form a complete preference data construction and optimization framework, which is used to effectively suppress illusions (reducing the illusion rate by 40-50%) and enhance image-text consistency through the bidirectional co-evolution of model policy and preference data distribution during visual language model training. It can still achieve excellent performance under low data volume conditions (about 3k preference data) and is suitable for alignment training of various visual language models.

[0158] Performance verification:

[0159] 1. The method described in this invention (D) 2 PO) was compared with other visual language alignment and preference optimization methods on the MMHalBench, ObjectHalBench, AMBER and POPE Adv benchmarks. The comparison results are shown in Tables 1 and 2. The other visual language alignment and preference optimization methods include mDPO, S-VCO, VCD, RLHF-V, V-DPO, POVID, OPA-DPO, MFPO, HSA-DPO, NaPO, HALVA, RLAIF-V, RoVRM, Octopus, LLaVA-1.5 and DPO.

[0160] Parameter settings: Batch size is 32; learning rate is 1×10⁻⁶. -6 The maximum text length is 128; the AdamW optimizer is used with weight decay of 0.05, gradient clipping of 1.0, and cosine learning rate decay (5% warm-up ratio); 8 responses are sampled for each image-question pair. (8); fusion coefficient 0.6, positive sample screening threshold 1.2, weighted discrimination The shrinkage strength coefficient increases linearly from 0.7 to 1.2. The regularization coefficient is 0.5. The weight is 0.2; segments with CLIP scores less than 0.7 or BERT scores less than 0.6 are labeled as negative samples; It is 0.3.

[0161] Table 1 Comparison results on MMHalBench and ObjectHalBench

[0162] ;

[0163] Table 2 Comparison results on AMBER and POPE Adv

[0164] ;

[0165] As can be seen, the method described in this invention achieves state-of-the-art performance on all benchmarks and with different parameter scales (7B and 13B) using only 3.1k data points, while other visual language alignment and preference optimization methods require 8-21K data points. Compared with traditional DPO and OPA-DPO, the method described in this invention uses less than 1 / 6 of the data points while achieving a significant performance improvement, highlighting its superior data efficiency and generalization ability across multimodal benchmarks.

[0166] 2. A detailed description of the same image was performed using the method described in this invention, along with LLaVA-1.5, traditional DPO, and OPA-DPO. The results are as follows: Figure 3 As shown, both LLaVA-1.5 and traditional DPO exhibit contextual illusions, such as fabricating non-existent background elements like trees, tourists, cars, or misinterpreting scene dynamics. While OPA-DPO reduces noise, it still introduces fictitious entities like pedestrians and busy streets. In contrast, the method described in this invention generates fine-grained descriptions, accurately identifying horses, red carriages, white boats, and surrounding objects, achieving a high degree of alignment with real visual content and exhibiting a low rate of illusion.

[0167] Example 2:

[0168] See Figure 4A multimodal preference optimization system based on bidirectional distribution alignment includes an on-policy candidate generation module, a positive and negative sample selection module, a bidirectional preference optimization module, and a model update module. The on-policy candidate generation module takes an original image, an edited image, and a question as input, and generates candidate response sets corresponding to the original image and the edited image under the current model policy. The positive and negative sample selection module, for each candidate response in the candidate response set, calculates its confidence score and reference consistency score with the reference response, selects positive samples from the candidate response set based on the confidence score and reference consistency score, and mines negative samples using entropy-guided methods. The strategy selects negative samples from the candidate response set. Specifically, the positive and negative sample selection module includes a positive sample selection module, an adaptive distribution shrinkage module, and an entropy-guided negative sample mining module. The positive sample selection module is used to select positive samples from the candidate response set based on the confidence score and reference consistency score of each candidate response, for each candidate response in the candidate response set. Specifically, the positive sample selection module is used to select positive samples based on the confidence score and reference consistency score according to the following steps: weighting the confidence score and the reference consistency score to obtain the overall quality score, and determining whether the overall quality score exceeds a preset threshold. If so, then the candidate response is taken as a positive sample. The positive sample screening module is used to calculate the confidence score according to the following formula:

[0169] ;

[0170] In the above formula, Candidate response Confidence score; For the input original image Editing images With the question In the current model strategy Candidate responses generated by downsampling , ; for The Middle One token; For the front A prefix consisting of tokens; Candidate response The sequence length;

[0171] The positive sample screening module is used to calculate the reference consistency score according to the following formula:

[0172] ;

[0173] In the above formula, Candidate response Reference Response Reference consistency score; This indicates that candidate responses are selected by embedding the encoder. Or refer to the response Mapped to a high-dimensional semantic vector; This represents the function for calculating cosine similarity.

[0174] The positive sample screening module is used to calculate the overall quality score according to the following formula:

[0175] ;

[0176] In the above formula, Candidate response The overall quality score; To control the relative weights of confidence scores and reference consistency scores, a fusion coefficient was used.

[0177] The adaptive distribution shrinking module is used to dynamically reconstruct the model strategy when positive samples cannot be obtained from the candidate response set corresponding to the original image or the candidate response set corresponding to the edited image. Under the reconstructed model strategy, an on-policy strategy is used to regenerate the candidate response set corresponding to the original image or the candidate response set corresponding to the edited image. The adaptive distribution shrinking strategy includes: firstly, calculating the sampling weight based on the mutual information gain of each candidate response to the reference response, the cross-candidate consistency among candidate responses in the candidate response set, and the length regularization term of each candidate response; then, obtaining the reconstructed model strategy based on the sampling weight. Specifically, the adaptive distribution shrinking module is used to calculate the mutual information gain of each candidate response to the reference response according to the following formula:

[0178] ;

[0179] In the above formula, For the candidate response set Each candidate response is relative to the reference response. Mutual information gain; For the first The edge entropy of each candidate response; In a given reference response Next The conditional entropy of each candidate response;

[0180] The adaptive distribution shrinking module is used to calculate cross-candidate consistency among candidate responses in the candidate response set according to the following formula:

[0181] ;

[0182] In the above formula, To ensure cross-candidate consistency among candidate responses within the candidate response set; The total number of candidate responses in the candidate response set; This indicates that the first encoder will be embedded. One candidate response or the One reference response Mapped to a high-dimensional semantic vector;

[0183] The adaptive distribution shrinking module is used to calculate the length regularization term for each candidate response according to the following formula:

[0184] ;

[0185] In the above formula, For the first The length regularization term for each candidate response; The regularization coefficient is used. For the first The token length of each candidate response; The average token length of all candidate responses in the candidate response set.

[0186] The adaptive distribution shrinkage module is used to calculate the sampling weights according to the following formula:

[0187] ;

[0188] In the above formula, For the first The sampling weights of each candidate response; For weighted discrimination;

[0189] The adaptive distribution shrinkage module is used to obtain the reconstructed model strategy based on the sampling weights according to the following formula:

[0190] ;

[0191] In the above formula, The reconstructed model strategy; This is the current model strategy; It is the shrinkage strength coefficient;

[0192] The entropy-guided negative sample mining module is used for candidate responses. ,in Given the token length, calculate the candidate response. The dynamic entropy threshold; for candidate responses The first in tokens Calculate the first tokens Predicted entropy; candidate responses Tokens whose predicted entropy exceeds the maximum threshold are marked as high-entropy fragments, while the remaining tokens are marked as low-entropy fragments. Then, the CLIP score of both high-entropy and low-entropy fragments is calculated with respect to the reference image for image-text consistency detection, and the BERT score is calculated with respect to the reference response for semantic consistency detection. If either a high-entropy or low-entropy fragment fails to meet either of these detection criteria, it is classified as a negative sample. Specifically, the entropy-guided negative sample mining module calculates the maximum threshold according to the following formula:

[0193] ;

[0194] In the above formula, Candidate response The dynamic entropy threshold; This is the mean of the predicted entropy of all tokens within the batch; The standard deviation of the predicted entropy for all tokens within the batch;

[0195] The first tokens The formula for calculating the prediction entropy is:

[0196] ;

[0197] In the above formula, Predicted entropy for each token; Indicates for the first tokens Previously, the model had generated predicted distributions for all historical token sequences; This indicates that all tokens have been traversed.

[0198] The bidirectional preference optimization module is used to input positive and negative samples selected from the candidate response set corresponding to the original image and positive and negative samples selected from the candidate response set corresponding to the edited image into the model, and calculate the contrast loss in both the image level and response level directions respectively; specifically, the bidirectional preference optimization module is used to calculate the contrast loss in both the image level and response level directions according to the following formula:

[0199] ;

[0200] ;

[0201] ;

[0202] ;

[0203] In the above formula, It is the sigmoid function; This is the preference scaling factor; To constrain the same problem The model processes the original image as input. Generated positive samples Compared to editing images Image-level contrast loss with higher preference scores; To constrain the same edited image Under the given conditions, the positive samples generated by the model Compared to the original image It has a higher preference score; To constrain the same original image Under these conditions, the model applies to positive samples. The preference is higher than that of negative samples Response level contrast loss; To edit the same image Under the given conditions, the positive samples generated by the model relative to negative samples Response level contrast loss; , These are positive and negative samples, respectively, selected from the candidate response set corresponding to the edited image. , These are positive and negative samples obtained from the candidate response set corresponding to the original image, respectively. Indicates the current model policy Below, given the problem and editing images Generate positive samples The probability of; Indicates a fixed model strategy Below, given the problem and editing images Generate positive samples The probability of; In the current model policy Below, given the problem and the original image Generate positive samples The probability of; For a fixed model strategy Below, given the problem and the original image Generate positive samples The probability of; Indicates the current model policy Below, given the problem and editing images Generate positive samples The probability of; Indicates a fixed model strategy Below, given the problem and editing images Generate positive samples The probability of; Indicates the current model policy Below, given the problem and the original image Generate negative samples The probability of; Indicates a fixed model strategy Below, given the problem and the original image Generate negative samples The probability of; Indicates the current model policy Below, given the problem and editing images Generate positive samples The probability of; Indicates a fixed model strategy Below, given the problem and editing images Generate positive samples The probability of; Indicates the current model policy Below, given the problem and editing images Generate negative samples The probability of; Indicates a fixed model strategy Below, given the problem and editing images Generate negative samples The probability of;

[0204] The model update module is used to calculate the overall model loss based on the contrast loss in both the image level and response level directions, and to update the model strategy based on the overall model loss until the model converges; specifically, the model update module is used to calculate the overall model loss according to the following formula:

[0205] ;

[0206] In the above formula, This represents the overall loss of the model. These are the weights used to balance the contrast loss in both the image-level and response-level directions.

[0207] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program goods. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program goods embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0208] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0209] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0210] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0211] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A multimodal preference optimization method based on bidirectional distribution alignment, characterized in that: The multimodal preference optimization method includes: S1. Based on the input original image, edited image, and question, generate a candidate response set corresponding to the original image and a candidate response set corresponding to the edited image under the current model strategy; S2. For each candidate response in the candidate response set, positive samples are selected from the candidate response set based on its confidence score and reference consistency score with the reference response, and negative samples are selected from the candidate response set through an entropy-guided negative sample mining strategy. S3. Input the positive and negative samples selected from the candidate response set corresponding to the original image and the positive and negative samples selected from the candidate response set corresponding to the edited image into the model, and calculate the contrast loss in both the image level and the response level directions respectively. S4. Calculate the overall model loss based on the contrast loss in both the image and response directions, and update the model strategy based on the overall model loss; repeat the above steps until the model converges.

2. The multimodal preference optimization method based on bidirectional distribution alignment according to claim 1, characterized in that: In S2, when positive samples cannot be obtained from the candidate response set corresponding to the original image or the candidate response set corresponding to the edited image, the model strategy is dynamically reconstructed through an adaptive distribution shrinkage strategy. Under the reconstructed model strategy, the on-policy strategy is used to regenerate the candidate response set corresponding to the original image or the candidate response set corresponding to the edited image. The adaptive distribution shrinkage strategy includes: firstly, calculating the sampling weight based on the mutual information gain of each candidate response to the reference response, the cross-candidate consistency among candidate responses in the candidate response set, and the length regularization term of each candidate response; and then obtaining the reconstructed model strategy based on the sampling weight.

3. The multimodal preference optimization method based on bidirectional distribution alignment according to claim 2, characterized in that: The formula for calculating the mutual information gain of each candidate response with respect to the reference response is as follows: ； In the above formula, For the candidate response set Each candidate response is relative to the reference response. Mutual information gain; For the first The edge entropy of each candidate response; In a given reference response Next The conditional entropy of each candidate response; The formula for calculating cross-candidate consistency among candidate responses in the candidate response set is as follows: ； In the above formula, To ensure cross-candidate consistency among candidate responses within the candidate response set; The total number of candidate responses in the candidate response set; This indicates that the first encoder will be embedded. One candidate response or the One reference response Mapped to a high-dimensional semantic vector; The formula for calculating the length regularization term of each candidate response is as follows: ； In the above formula, For the first The length regularization term for each candidate response; The regularization coefficient is used. For the first The token length of each candidate response; The average token length of all candidate responses in the candidate response set.

4. The multimodal preference optimization method based on bidirectional distribution alignment according to claim 3, characterized in that: The formula for calculating the sampling weight is: ； In the above formula, For the first The sampling weights of each candidate response; For weighted discrimination; The reconstructed model strategy is obtained based on the sampling weights using the following formula: ； In the above formula, For the reconstructed model strategy Below, given image and the problem Generate response The probability of; In the current model policy Below, given image and the problem Generate response The probability of; , For the original image, For editing images; This is the shrinkage strength coefficient.

5. A multimodal preference optimization method based on bidirectional distribution alignment according to any one of claims 1-4, characterized in that: In S2, the entropy-guided negative sample mining strategy includes: for candidate responses ,in Given the token length, calculate the candidate response. The dynamic entropy threshold; for candidate responses The first in tokens Calculate the first tokens Predicted entropy; candidate responses Tokens whose predicted entropy exceeds the maximum threshold are marked as high-entropy fragments, while the remaining tokens are marked as low-entropy fragments. Then, the CLIP score of the obtained high-entropy and low-entropy fragments with the reference image is calculated for image-text consistency detection, and the BERT score of the fragments with the reference response is calculated for semantic consistency detection. If a high-entropy fragment or a low-entropy fragment fails to meet either of these detection criteria, it is determined as a negative sample.

6. The multimodal preference optimization method based on bidirectional distribution alignment according to claim 5, characterized in that: The formula for calculating the maximum threshold is: ； In the above formula, Candidate response The dynamic entropy threshold; This is the mean of the predicted entropy of all tokens within the batch; The standard deviation of the predicted entropy for all tokens within the batch; The first tokens The formula for calculating the prediction entropy is: ； In the above formula, Predicted entropy for each token; Indicates for the first tokens Previously, the model had generated predicted distributions for all historical token sequences; This indicates that all tokens are iterated over.

7. A multimodal preference optimization method based on bidirectional distribution alignment according to any one of claims 1-4, characterized in that: In S2, the step of selecting positive samples based on confidence scores and reference consistency scores includes: weighting the confidence scores and reference consistency scores to obtain an overall quality score, and determining whether the overall quality score exceeds a preset threshold. If so, then the candidate response is taken as a positive sample. .

8. The multimodal preference optimization method based on bidirectional distribution alignment according to claim 7, characterized in that: Calculate the confidence score using the following formula: ； In the above formula, Represents a given image and the problem Candidate responses generated by sampling; , For the original image, For editing images; Candidate response Confidence score; for The Middle One token; For the front A prefix consisting of tokens; Candidate response The sequence length; The reference consistency score is calculated using the following formula: ； In the above formula, Candidate response Reference Response Reference consistency score; This indicates that candidate responses are selected by embedding the encoder. Or refer to the response Mapped to a high-dimensional semantic vector; This represents the function for calculating cosine similarity. The formula for calculating the overall quality score is as follows: ； In the above formula, Candidate response The overall quality score; To control the relative weights of confidence scores and reference consistency scores, a fusion coefficient is used.

9. A multimodal preference optimization method based on bidirectional distribution alignment according to any one of claims 1-4, characterized in that: In S4, the formula for calculating the overall loss of the model is: ； ； ； ； ； In the above formula, This represents the overall loss of the model. These are the weights used to balance the contrast loss in both the image-level and response-level directions; It is the sigmoid function; This is the preference scaling factor; To constrain the same problem The model processes the original image as input. Generated positive samples Compared to editing images Image-level contrast loss with higher preference scores; To constrain the same edited image Under the given conditions, the positive samples generated by the model Compared to the original image It has a higher preference score; To constrain the same original image Under these conditions, the model applies to positive samples. The preference is higher than that of negative samples Response level contrast loss; To edit the same image Under the given conditions, the positive samples generated by the model relative to negative samples Response level contrast loss; , These are positive and negative samples, respectively, selected from the candidate response set corresponding to the edited image. , These are positive and negative samples obtained from the candidate response set corresponding to the original image, respectively. Indicates the current model policy Below, given the problem and editing images Generate positive samples The probability of; Indicates a fixed model strategy Below, given the problem and editing images Generate positive samples The probability of; In the current model policy Below, given the problem and the original image Generate positive samples The probability of; For a fixed model strategy Below, given the problem and the original image Generate positive samples The probability of; Indicates the current model policy Below, given the problem and editing images Generate positive samples The probability of; Indicates a fixed model strategy Below, given the problem and editing images Generate positive samples The probability of; Indicates the current model policy Below, given the problem and the original image Generate negative samples The probability of; Indicates a fixed model strategy Below, given the problem and the original image Generate negative samples The probability of; Indicates the current model policy Below, given the problem and editing images Generate positive samples The probability of; Indicates a fixed model strategy Below, given the problem and editing images Generate positive samples The probability of; Indicates the current model policy Below, given the problem and editing images Generate negative samples The probability of; Indicates a fixed model strategy Below, given the problem and editing images Generate negative samples The probability of.

10. A multimodal preference optimization system based on bidirectional distributed alignment, characterized in that: The multimodal preference optimization system is based on the multimodal preference optimization method of claim 1, and the system includes: The on-policy candidate generation module is used to take the original image, the edited image, and the question as input, and generate a candidate response set corresponding to the original image and a candidate response set corresponding to the edited image under the current model policy. The positive and negative sample screening module is used to select positive samples from the candidate response set based on the confidence score and the reference consistency score with the reference response for each candidate response in the candidate response set, and to select negative samples from the candidate response set through an entropy-guided negative sample mining strategy. The bidirectional preference optimization module is used to input positive and negative samples selected from the candidate response set corresponding to the original image and positive and negative samples selected from the candidate response set corresponding to the edited image into the model, and calculate the contrast loss in both the image level and the response level. The model update module is used to calculate the overall model loss based on the contrast loss in both the image level and response level directions, and update the model strategy based on the overall model loss until the model converges.

Images