Visual language model bidirectional preference optimization method and system based on label level supervision

Abstract

The invention belongs to the technical field of visual language models, and particularly relates to a visual language model bidirectional preference optimization method and system based on marker level supervision, and the method comprises the steps: generating a comparison image of an original image according to a comparison generation strategy and a similarity filtering mechanism, so as to construct a BiDPO data set; then constructing a bidirectional sequence level loss function based on a bidirectional preference optimization strategy to enable the model to align an image-answer pair in an original direction and a reverse direction; marking the answers, and constructing a bidirectional mark-level regularization loss function based on mark-level rewards and regularization, so that marks with positive rewards are added to the model, and marks with negative rewards are reduced; constructing a total loss function based on the bidirectional sequence level loss function and the bidirectional mark level regularization loss function; and finally, based on the BiDPO data set, carrying out bidirectional training on the visual language model by minimizing a total loss function. According to the method, the illusion generation rate can be remarkably reduced, and the overall visual language performance is comprehensively improved.

Description

Technical Field

[0001] This invention belongs to the field of visual language model technology, specifically relating to a method, system, device, and medium for bidirectional preference optimization of visual language models based on label-level supervision. Background Technology

[0002] In recent years, visual language models (VLMs) have made significant progress in image-text reasoning and understanding. However, these models often rely excessively on prior linguistic knowledge and have low sensitivity to fine visual details, leading to visual illusions in predictions. These illusions affect the reliability of VLMs and limit their application in safety-critical fields such as medical imaging and autonomous driving. To address this challenge, existing research has explored contrastive decoding, over-trust penalties, and more robust baseline strategies. However, current visual language models still lack a label-level alignment mechanism between vision and language, which limits their effectiveness in scenarios requiring precise detail understanding.

[0003] Preference optimization (DPO) has become a key paradigm for aligning large language models (LLMs) with human values ​​and intentions. DPO is a preference optimization method that matches LLMs to human preferences without requiring an explicit reward function. By refactoring the preference alignment problem into a supervised learning task, it achieves more stable and sample-efficient training even in the presence of noise or imperfect feedback. Existing research has attempted to apply DPO to the visual-language domain. mDPO introduces visual perturbations or visually guided preference objectives to enhance attention to visual content and reduce hallucinations. However, these methods typically construct coarse negative samples by adding low-level perturbations (such as random cropping or noise injection). These perturbations are often semantically irrelevant to the task and rely primarily on coarse-grained visual supervision, making it difficult to capture fine-grained visual-linguistic semantic differences in visual scenes. This hinders robust label-level localization, leading to shortcut learning phenomena where the model either ignores image features or overfits surface visual cues, ultimately weakening the visual-linguistic alignment effect. Summary of the Invention

[0004] The purpose of this invention is to address the aforementioned problems in the existing technology by providing a label-level supervised bidirectional preference optimization method, system, device, and medium for visual language models. This method trains the model using a bidirectional preference optimization strategy, enabling the model to correctly distinguish image-answer pairings in both the forward and reverse directions, thereby enhancing semantic consistency, reducing reliance on false cues, and preventing visual illusion problems in the model.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] Firstly, this invention proposes a bidirectional preference optimization method for visual language models based on label-level supervision, the bidirectional preference optimization method for visual language models comprising:

[0007] S1. Construct a contrast generation strategy and a similarity filtering mechanism. Based on the contrast generation strategy and the similarity filtering mechanism, generate a contrast image that maintains global semantic consistency with the original image while having visual feature differences. Construct a BiDPO dataset for bidirectional preference optimization based on the original image and the contrast image.

[0008] S2. Construct a bidirectional sequence-level loss function based on a bidirectional preference optimization strategy to align the image-answer pairs in the original direction and the reverse direction; label the answers, and construct a bidirectional label-level regularization loss function based on label-level rewards and regularization to increase labels with positive rewards and decrease labels with negative rewards; construct a total loss function based on the bidirectional sequence-level loss function and the bidirectional label-level regularization loss function.

[0009] S3. Based on the BiDPO dataset, the visual language model is trained bidirectionally by minimizing the total loss function, and the trained visual language model is used for testing and calculation.

[0010] S1 includes:

[0011] S11. Extract the original image ,question and the original image Regarding the question The correct answer In response to the problem A rule-based annotation strategy is used to analyze the semantic focus, and based on the identified semantic focus, an original image is generated through a comparative generation strategy. Comparison images and comparison images Regarding the question Comparison of answers Comparison Images With the original image There are local visual differences;

[0012] S12, Transfer the original image Comparison Image Forming candidate pairings For each candidate pair Calculate their global similarity and retain only candidate pairs whose global similarity is within a preset range;

[0013] S13. Based on the retained candidate pairings, a BiDPO dataset is finally generated. Each pairing in the BiDPO dataset is represented in the following combination form: ;in , Each represents a set of questions Semantic matching of images and answer pairs.

[0014] The global similarity refers to CLIP similarity, with a preset range of [0.3, 0.9].

[0015] The comparison generation strategy includes object replacement, attribute modification, quantity adjustment, or position flipping.

[0016] In step S2, the steps for constructing the bidirectional sequence-level loss function include:

[0017] For the original direction By requiring the correct answer For the original image The preference is better than the comparison answer. To define the sequence-level loss function for the original direction:

[0018] ;

[0019] In the above formula, This represents the sigmoid function. It's a temperature over-parameter. Representative strategy model, Represents the reference model;

[0020] For the reverse By requiring comparison of answers Comparison Images The preference is better than the correct answer To define the reverse sequence-level loss function:

[0021] .

[0022] In step S2, the construction steps of the bidirectional label-level regularization loss function include:

[0023] For a length of Each marker in the correct answer sequence The correct answer is found by comparing its original image. Comparison Image The log-likelihood value is used to calculate the bidirectional labeled level reward, and the bidirectional labeled level regularization loss function is calculated based on the bidirectional labeled level reward; the expression for the bidirectional labeled level reward is:

[0024] ;

[0025] ;

[0026] In the above formula, For the original image With the question Under the condition of the first One answer Two-way tag-level reward value; To compare images With the question Under the condition of the first One answer Two-way tag-level reward value; For the original image With the question Given the prefix answer under the given conditions The reference model generates the next answer. The probability of; To compare images With the question Given the prefix answer under the given conditions The reference model generates the next answer. The probability of; To compare images With the question Given the prefix answer under the given conditions The reference model generates the next answer. The probability of; For the original image With the question Given the prefix answer under the given conditions The reference model generates the next answer. The probability of; This indicates normalization processing;

[0027] The expression for the bidirectional label-level regularization loss function is:

[0028]

[0029] ;

[0030] In the above formula, , These represent the label-level regularization loss functions for the original direction and the reverse direction, respectively; For the original image With the question Given the prefix answer under the given conditions The strategy model generates the next answer. The probability of; To compare images With the question Given the prefix answer under the given conditions The strategy model generates the next answer. The probability of.

[0031] The expression for the total loss function is:

[0032] ;

[0033] In the above formula, Represents the total loss function; , These are the sequence-level loss functions for the original direction and the reverse direction, respectively; , These represent the label-level regularization loss functions for the original direction and the reverse direction, respectively; For balancing parameters.

[0034] Secondly, this invention proposes a bidirectional preference optimization system for visual language models based on label-level supervision, the bidirectional preference optimization system for visual language models comprising:

[0035] The dataset construction module is used to build a contrast generation strategy and a similarity filtering mechanism. Based on the contrast generation strategy and the similarity filtering mechanism, a contrast image is generated that maintains global semantic consistency with the original image while having visual feature differences. Based on the original image and the contrast image, a BiDPO dataset for bidirectional preference optimization is constructed.

[0036] The loss function construction module is used to construct a bidirectional sequence-level loss function based on a bidirectional preference optimization strategy to align the image-answer pairs in the original direction and the reverse direction; to label the answers, a bidirectional label-level regularization loss function is constructed based on label-level rewards and regularization, so that the model increases labels with positive rewards and reduces labels with negative rewards; and a total loss function is constructed based on the bidirectional sequence-level loss function and the bidirectional label-level regularization loss function.

[0037] The model training module is used to train the visual language model bidirectionally based on the BiDPO dataset by minimizing the total loss function, and then use the trained visual language model for testing.

[0038] The dataset building module is used to build a BiDPO dataset for bidirectional preference optimization according to the following steps:

[0039] S11. Extract the original image ,question and the original image Regarding the question The correct answer In response to the problem A rule-based annotation strategy is used to analyze the semantic focus, and based on the identified semantic focus, an original image is generated through a comparative generation strategy. Comparison images and comparison images Regarding the question Comparison of answers Comparison Images With the original image There are local visual differences;

[0040] S12, Transfer the original image Comparison Image Forming candidate pairings For each candidate pair Calculate their global similarity and retain only candidate pairs whose global similarity is within a preset range;

[0041] S13. Based on the retained candidate pairings, a BiDPO dataset is finally generated. Each pairing in the BiDPO dataset is represented in the following combination form: ;in , Each represents a set of questions Semantic matching of images and answer pairs.

[0042] The global similarity refers to CLIP similarity, with a preset range of [0.3, 0.9].

[0043] The comparison generation strategy includes object replacement, attribute modification, quantity adjustment, or position flipping.

[0044] The loss function construction module is used to construct a bidirectional sequence-level loss function according to the following steps:

[0045] For the original direction By requiring the correct answer For the original image The preference is better than the comparison answer. To define the sequence-level loss function for the original direction:

[0046] ;

[0047] In the above formula, This represents the sigmoid function. It's a temperature over-parameter. Representative strategy model, Represents the reference model;

[0048] For the reverse By requiring comparison of answers Comparison Images The preference is better than the correct answer To define the reverse sequence-level loss function:

[0049] .

[0050] The loss function construction module is used to construct a bidirectional label-level regularization loss function according to the following steps:

[0051] For a length of Each marker in the correct answer sequence The correct answer is found by comparing its original image. Comparison Image The log-likelihood value is used to calculate the bidirectional labeled level reward, and the bidirectional labeled level regularization loss function is calculated based on the bidirectional labeled level reward; the expression for the bidirectional labeled level reward is:

[0052] ;

[0053] ;

[0054] In the above formula, For the original image With the question Under the condition of the first One answer Two-way tag-level reward value; To compare images With the question Under the condition of the first One answer Two-way tag-level reward value; For the original image With the question Given the prefix answer under the given conditions The reference model generates the next answer. The probability of; To compare images With the question Given the prefix answer under the given conditions The reference model generates the next answer. The probability of; To compare images With the question Given the prefix answer under the given conditions The reference model generates the next answer. The probability of; For the original image With the question Given the prefix answer under the given conditions The reference model generates the next answer. The probability of; This indicates normalization processing;

[0055] The expression for the bidirectional label-level regularization loss function is:

[0056]

[0057] ;

[0058] In the above formula, , These represent the label-level regularization loss functions for the original direction and the reverse direction, respectively; For the original image With the question Given the prefix answer under the given conditions The strategy model generates the next answer. The probability of; To compare images With the question Given the prefix answer under the given conditions The strategy model generates the next answer. The probability of.

[0059] The loss function construction module is used to construct the total loss function according to the following steps:

[0060] ;

[0061] In the above formula, Represents the total loss function; , These are the sequence-level loss functions for the original direction and the reverse direction, respectively; , These represent the label-level regularization loss functions for the original direction and the reverse direction, respectively; For balancing parameters.

[0062] Thirdly, the present invention proposes a visual language model bidirectional preference optimization device based on label-level supervision. The visual language model bidirectional preference optimization device includes a memory and a processor. The memory is used to store computer program code and transmit the computer program code to the processor. The processor is used to execute the aforementioned visual language model bidirectional preference optimization method according to the instructions in the computer program code.

[0063] Fourthly, the present invention proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the aforementioned visual language model bidirectional preference optimization method.

[0064] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0065] 1. In the label-level supervised visual language model bidirectional preference optimization method described in this invention, when the original image and the question are input, the semantic focus of the question is first identified. A contrast generation strategy is used to generate a contrast image with subtle semantic conflicts. A similarity metric is used to ensure that the contrast image maintains global semantic consistency with the original image while introducing local visual differences. This design can achieve fine granularity while ensuring meaningful visual perturbations, and has the effectiveness of semantic contrast, thus providing support for subsequent preference optimization. Subsequently, a bidirectional preference optimization mechanism is further introduced during model training. The model is trained to distinguish the correct image-answer pairing in both the forward and reverse directions, requiring the correct answer to be presented first in the corresponding original image, while avoiding conflicts with the contrast image. This guides the model to focus on key visual cues, thereby enhancing semantic consistency and reducing dependence on false cues, ultimately reducing illusions and improving visual localization accuracy.

[0066] 2. The label-level supervised visual language model bidirectional preference optimization method described in this invention integrates label-level alignment and regularization strategies, guiding the model to associate each output label with relevant visual evidence and improving the visual language semantic alignment capability. When the model-generated answer has higher support in the original image than in the comparison image, the model is rewarded. This design prompts the model to extract key visual evidence for each label and promotes fine-grained visual language alignment and robustness, further reducing hallucination phenomena. Attached Figure Description

[0067] Figure 1 This is a flowchart of the bidirectional preference optimization method for visual language models described in this invention.

[0068] Figure 2 This is a schematic diagram illustrating the principle of the bidirectional preference optimization method for visual language models described in this invention.

[0069] Figure 3 This is a schematic diagram illustrating object replacement, attribute modification, quantity adjustment, and position flipping in the comparison generation strategy described in this invention.

[0070] Figure 4 This is a schematic diagram of the structure of the visual language model bidirectional preference optimization system described in this invention.

[0071] Figure 5 This is a schematic diagram of the structure of the visual language model bidirectional preference optimization device described in this invention. Detailed Implementation

[0072] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings.

[0073] Example 1:

[0074] See Figure 1 , Figure 2 A bidirectional preference optimization method for visual language models based on label-level supervision (BiDPO model for short) includes the following steps:

[0075] S1. Construct a contrast generation strategy and a similarity filtering mechanism. Based on the contrast generation strategy and the similarity filtering mechanism, generate contrast images that maintain global semantic consistency with the original images while having visual feature differences. Construct a BiDPO dataset for bidirectional preference optimization based on the original images and contrast images.

[0076] Specifically, the steps for constructing the BiDPO dataset include:

[0077] S11. Extract the original image ,question and the original image Regarding the question The correct answer In response to the problem A rule-based annotation strategy is used to analyze the semantic focus, and based on the identified semantic focus, an original image is generated through a comparative generation strategy. Comparison images and comparison images Regarding the question Comparison of answers Comparison Images With the original image There are local visual differences;

[0078] like Figure 3 As shown, the contrast generation strategy includes object replacement, attribute modification, quantity adjustment, or position flipping; for each task, a prompt describing the minimum required task-related visual changes is generated, which is compared with the original image. Both are fed into the GPT-4o visual language model to generate contrasting images with specific modifications (such as object replacement in the contrast generation strategy). and their corresponding answers For example, object replacement can replace the object type in the original image, attribute modification can modify the color of the object in the original image, quantity adjustment can modify the quantity of the object in the original image, and position flipping can flip the position of the object in the original image.

[0079] S12, Transfer the original image Comparison Image Forming candidate pairings For each candidate pair Calculate the global similarity and retain only candidate pairs whose global similarity falls within a preset range; preferably, the global similarity refers to CLIP similarity, with a preset range of [0.3, 0.9]; reasonably setting the preset range of global similarity ensures that the original image... Comparison Image Maintaining overall semantic consistency while highlighting local visual differences between the two; for each candidate pairing The similarity filtering mechanism is to calculate the global similarity and retain only candidate pairs whose global similarity is within a preset range.

[0080] S13. Based on the retained candidate pairings, a BiDPO dataset is finally generated. Each pairing in the BiDPO dataset is represented in the following combination form: ;in , Each represents a set of questions Semantic matching of images and answer pairs.

[0081] S2. Construct a bidirectional sequence-level loss function based on a bidirectional preference optimization strategy to align the image-answer pairs in the original direction and the reverse direction; label the answers, and construct a bidirectional label-level regularization loss function based on label-level rewards and regularization to increase labels with positive rewards and decrease labels with negative rewards; construct a total loss function based on the bidirectional sequence-level loss function and the bidirectional label-level regularization loss function.

[0082] Specifically, the construction steps of the bidirectional sequence-level loss function include:

[0083] For the original direction By requiring the correct answer With the original image Its compatibility is better than the compared answer. To define the sequence-level loss function for the original direction:

[0084] ;

[0085] In the above formula, This represents the sigmoid function. It's a temperature over-parameter. Representative strategy model, The reference model represents the policy model, which is the visual language model to be optimized.

[0086] For the reverse By requiring comparison of answers Comparison Image The compatibility is better than the correct answer. To define the reverse sequence-level loss function:

[0087] .

[0088] This bidirectional sequence-level loss function prevents shortcut learning by requiring the model to align the correct image-answer pairs in both the original and reverse directions. While the bidirectional sequence-level loss function encourages the model to choose the answer that matches the corresponding image, it may overlook fine-grained errors in the generated sequence. To address this issue, this invention introduces a label-level supervision mechanism that includes label-level rewards and regularization, thereby achieving more accurate performance allocation.

[0089] Specifically, the construction steps of the bidirectional label-level regularization loss function include:

[0090] For each marker in the correct answer sequence of length T The correct answer is found by comparing its original image. Comparison Image The log-likelihood value is used to calculate the bidirectional labeled level reward, and the bidirectional labeled level regularization loss function is calculated based on the bidirectional labeled level reward; the expression for the bidirectional labeled level reward is:

[0091] ;

[0092] ;

[0093] In the above formula, For the original image With the question Under the condition of the first One answer Two-way tag-level reward value; To compare images With the question Under the condition of the first One answer Two-way tag-level reward value; For the original image With the question Given the prefix answer under the given conditions The reference model generates the next answer. The probability of; To compare images With the question Given the prefix answer under the given conditions The reference model generates the next answer. The probability of; To compare images With the question Given the prefix answer under the given conditions The reference model generates the next answer. The probability of; For the original image With the question Given the prefix answer under the given conditions The reference model generates the next answer. The probability of; This indicates normalization processing; the label-level reward reflects the increase in the probability of a label appearing in the corresponding correct image compared to the comparison image.

[0094] The expression for the bidirectional label-level regularization loss function is:

[0095] ;

[0096] ;

[0097] In the above formula, , These represent the label-level regularization loss functions for the original direction and the reverse direction, respectively; For the original image With the question Given the prefix answer under the given conditions The strategy model generates the next answer. The probability of; To compare images With the question Given the prefix answer under the given conditions The strategy model generates the next answer. The probability of.

[0098] Regularization can encourage the model to increase the likelihood of labels with positive label level rewards and decrease the likelihood of labels with negative rewards, thereby improving label level alignment.

[0099] The expression for the total loss function is:

[0100] ;

[0101] In the above formula, Represents the total loss function; The above total loss function is used to balance the parameters. It allows the model to benefit from both sequence-level preference supervision and label-level regularization, thereby enhancing the alignment and robustness of VLMs models.

[0102] S3. Based on the BiDPO dataset, the visual language model is trained bidirectionally by minimizing the total loss function, and the trained model is then used for testing. The policy network is the visual language model to be trained, and its parameters can be initialized from a large-scale pre-trained model. The reference network is fixed as a frozen model with the same structure as the policy network, and its parameters remain unchanged during training to provide a baseline probability distribution. The policy network continuously updates its parameters through backpropagation under the guidance of the reward signal to minimize the bidirectional loss function.

[0103] Performance verification:

[0104] 1. The BiDPO model described in this invention is compared with other visual language alignment and preference optimization models, including mDPO, S-VCO, VCD, RLAIF-V, V-DPO, POVID, Octopus, RoVRM, OPA-DPO, and MFPO. During training, the BiDPO model of this invention constructs a BiDPO dataset containing 12,000 samples. The samples in the BiDPO dataset are automatically generated from the LLaVAInstruct-150K dataset using the GPT-4o model. During generation, targeted visual modifications are made based on the semantic focus of the problem, followed by CLIP similarity filtering to ensure semantic consistency and achieve effective fine-grained supervision. The model performance is evaluated based on a series of visual language test scenarios (MMHalBench, Object HalBench, and AMBER) and metrics (↓ indicates lower values ​​are better). The evaluation results are shown in Table 1.

[0105] Table 1 Performance comparison of the BiDPO model with other models

[0106]

[0107] Table 1 shows the test results of the BiDPO model described in this invention on MMHalBench, ObjectHalBench, and AMBER. It can be seen that the BiDPO model exhibits the lowest hallucination rate in all test scenarios, indicating that the BiDPO model significantly reduces the hallucination generation rate, while achieving a comprehensive improvement in overall visual language performance across all evaluation metrics.

[0108] 2. The impact of key components in the BiDPO model was tested through ablation experiments. The hallucination rates on MMHalBench and AMBER are shown in Table 2. It can be seen that removing the bidirectional preference optimization mechanism significantly increases the hallucination rate on both the MMHalBench and AMBER test sets. Similarly, removing the marker-level alignment and regularization strategy also degrades performance. The complete BiDPO model exhibits the lowest hallucination rate, demonstrating the complementary effect between the bidirectional preference optimization mechanism and the marker-level alignment and regularization strategy.

[0109] Table 2 Ablation Experiment Results

[0110]

[0111] 3. Testing the impact of different CLIP similarity preset ranges. The hallucination rates on MMHalBench and AMBER are shown in Table 3. It can be seen that the similarity range of [0.3, 0.9] has the best performance. Whether CLIP similarity filtering is completely disabled or an overly strict or lenient similarity range is used, the hallucination rate will increase. This indicates that using an appropriate similarity range in the dataset construction can balance semantic consistency and visual feature differences.

[0112] Table 3. The Influence of Different CLIP Similarity Preset Ranges

[0113]

[0114] Example 2:

[0115] See Figure 4 A label-level supervised visual language model bidirectional preference optimization system is disclosed, comprising a dataset construction module, a loss function construction module, and a model training module. The dataset construction module is used to construct a contrast generation strategy and a similarity filtering mechanism. Based on the contrast generation strategy and the similarity filtering mechanism, contrast images that maintain global semantic consistency with the original images while exhibiting visual feature differences are generated. A BiDPO dataset for bidirectional preference optimization is constructed based on the original images and the contrast images. Specifically, the dataset construction module constructs the BiDPO dataset for bidirectional preference optimization according to the following steps:

[0116] S11. Extract the original image ,question and the original image Regarding the question The correct answer In response to the problem A rule-based annotation strategy is used to analyze the semantic focus, and based on the identified semantic focus, an original image is generated through a comparative generation strategy. Comparison images and comparison images Regarding the question Comparison of answers Comparison Images With the original image There are local visual differences; the contrast generation strategy mentioned above includes object replacement, attribute modification, quantity adjustment, or position flipping.

[0117] S12, Transfer the original image Comparison Image Forming candidate pairings For each candidate pair Calculate their global similarity and retain only candidate pairs whose global similarity is within a preset range; S13. Generate the BiDPO dataset based on the retained candidate pairs. Each pair in the BiDPO dataset is represented in the following combination form: ;in , Each represents a set of questions Image-answer pairs with semantic matching; wherein the global similarity refers to CLIP similarity, preferably within the preset range of [0.3, 0.9].

[0118] The loss function construction module is used to construct a bidirectional sequence-level loss function based on a bidirectional preference optimization strategy to align the image-answer pairs in the original direction and the reverse direction; to label the answers, and to construct a bidirectional label-level regularization loss function based on label-level rewards and regularization, so that the model increases labels with positive rewards and reduces labels with negative rewards; and to construct a total loss function based on the bidirectional sequence-level loss function and the bidirectional label-level regularization loss function.

[0119] Specifically, the loss function construction module is used to construct a bidirectional sequence-level loss function according to the following steps:

[0120] For the original direction By requiring the correct answer For the original image The preference is better than the comparison answer. To define the sequence-level loss function for the original direction:

[0121] ;

[0122] In the above formula, This represents the sigmoid function. It's a temperature over-parameter. Representative strategy model, Represents the reference model;

[0123] For the reverse By requiring comparison of answers Comparison Images The preference is better than the correct answer To define the reverse sequence-level loss function:

[0124] .

[0125] The loss function construction module is used to construct a bidirectional label-level regularization loss function according to the following steps:

[0126] For a length of Each marker in the correct answer sequence The correct answer is found by comparing its original image. Comparison Image The log-likelihood value is used to calculate the bidirectional labeled level reward, and the bidirectional labeled level regularization loss function is calculated based on the bidirectional labeled level reward; the expression for the bidirectional labeled level reward is:

[0127] ;

[0128] ;

[0129] In the above formula, For the original image With the question Under the condition of the first One answer Two-way tag-level reward value; To compare images With the question Under the condition of the first One answer Two-way tag-level reward value; For the original image With the question Given the prefix answer under the given conditions The reference model generates the next answer. The probability of; To compare images With the question Given the prefix answer under the given conditions The reference model generates the next answer. The probability of; To compare images With the question Given the prefix answer under the given conditions The reference model generates the next answer. The probability of; For the original image With the question Given the prefix answer under the given conditions The reference model generates the next answer. The probability of; This indicates normalization processing;

[0130] The expression for the bidirectional label-level regularization loss function is:

[0131]

[0132] ;

[0133] In the above formula, , These represent the label-level regularization loss functions for the original direction and the reverse direction, respectively; For the original image With the question Given the prefix answer under the given conditions The strategy model generates the next answer. The probability of; To compare images With the question Given the prefix answer under the given conditions The strategy model generates the next answer. The probability of.

[0134] The loss function construction module is used to construct the total loss function according to the following steps:

[0135] ;

[0136] In the above formula, Represents the total loss function; For balancing parameters.

[0137] The model training module is used to perform bidirectional training of the visual language model based on the BiDPO dataset by minimizing the total loss function, and to perform test calculations using the trained visual language model.

[0138] Example 3:

[0139] See Figure 5 A device for bidirectional preference optimization of a visual language model based on label-level supervision includes a memory and a processor; the memory is used to store computer program code and transmit the computer program code to the processor; the processor is used to execute the bidirectional preference optimization method of the visual language model described in Embodiment 1 according to the instructions in the computer program code.

[0140] Example 4:

[0141] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the visual language model bidirectional preference optimization method described in Embodiment 1.

[0142] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program goods. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program goods embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0143] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0144] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0145] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A bidirectional preference optimization method for visual language models based on label-level supervision, characterized by: The bidirectional preference optimization method for the visual language model includes: S1. Construct a contrast generation strategy and a similarity filtering mechanism. Based on the contrast generation strategy and the similarity filtering mechanism, generate a contrast image that maintains global semantic consistency with the original image while having visual feature differences. Construct a BiDPO dataset for bidirectional preference optimization based on the original image and the contrast image. S2. Construct a bidirectional sequence-level loss function based on a bidirectional preference optimization strategy to align the image-answer pairs in the original direction and the reverse direction; label the answers, and construct a bidirectional label-level regularization loss function based on label-level rewards and regularization to increase labels with positive rewards and decrease labels with negative rewards; construct a total loss function based on the bidirectional sequence-level loss function and the bidirectional label-level regularization loss function. S3. Based on the BiDPO dataset, the visual language model is trained bidirectionally by minimizing the total loss function, and the trained visual language model is used for testing and calculation.

2. The bidirectional preference optimization method for visual language models based on label-level supervision according to claim 1, characterized in that: S1 includes: S11. Extract the original image ,question and the original image Regarding the question The correct answer In response to the problem A rule-based annotation strategy is used to analyze the semantic focus, and based on the identified semantic focus, an original image is generated through a comparative generation strategy. Comparison images and comparison images Regarding the question Comparison of answers Comparison Images With the original image There are local visual differences; S12, Transfer the original image Comparison Image Forming candidate pairings For each candidate pair Calculate their global similarity and retain only candidate pairs whose global similarity is within a preset range; S13. Based on the retained candidate pairings, a BiDPO dataset is finally generated. Each pairing in the BiDPO dataset is represented in the following combination form: ;in , Each represents a set of questions Semantic matching of images and answer pairs.

3. The bidirectional preference optimization method for visual language models based on label-level supervision according to claim 2, characterized in that: The global similarity refers to CLIP similarity, with a preset range of [0.3, 0.9].

4. The bidirectional preference optimization method for visual language models based on label-level supervision according to claim 1, characterized in that: The comparison generation strategy includes object replacement, attribute modification, quantity adjustment, or position flipping.

5. The bidirectional preference optimization method for visual language models based on label-level supervision according to claim 1, characterized in that: In step S2, the steps for constructing the bidirectional sequence-level loss function include: For the original direction By requiring the correct answer With the original image Its compatibility is better than the compared answer. To define the sequence-level loss function for the original direction: ； In the above formula, This represents the sigmoid function. It's a temperature over-parameter. Representative strategy model, Represents the reference model; For the reverse By requiring comparison of answers Comparison Image The compatibility is better than the correct answer. To define the reverse sequence-level loss function: 。 6. The bidirectional preference optimization method for visual language models based on label-level supervision according to claim 1, characterized in that: In step S2, the construction steps of the bidirectional label-level regularization loss function include: For a length of Each marker in the correct answer sequence The correct answer is found by comparing its original image. Comparison Image The log-likelihood value is used to calculate the bidirectional labeled level reward, and the bidirectional labeled level regularization loss function is calculated based on the bidirectional labeled level reward; the expression for the bidirectional labeled level reward is: ； ； In the above formula, For the original image With the question Under the condition of the first One answer Two-way tag-level reward value; To compare images With the question Under the condition of the first One answer Two-way tag-level reward value; For the original image With the question Given the prefix answer under the given conditions The reference model generates the next answer. The probability of; To compare images With the question Given the prefix answer under the given conditions The reference model generates the next answer. The probability of; To compare images With the question Given the prefix answer under the given conditions The reference model generates the next answer. The probability of; For the original image With the question Given the prefix answer under the given conditions The reference model generates the next answer. The probability of; This indicates normalization processing; The expression for the bidirectional label-level regularization loss function is: ； ； In the above formula, , These represent the label-level regularization loss functions for the original direction and the reverse direction, respectively; For the original image With the question Given the prefix answer under the given conditions The strategy model generates the next answer. The probability of; To compare images With the question Given the prefix answer under the given conditions The strategy model generates the next answer. The probability of.

7. The bidirectional preference optimization method for visual language models based on label-level supervision according to claim 1, characterized in that: The expression for the total loss function is: ； In the above formula, Represents the total loss function; , These are the sequence-level loss functions for the original direction and the reverse direction, respectively; , These represent the label-level regularization loss functions for the original direction and the reverse direction, respectively; For balancing parameters.

8. A bidirectional preference optimization system for visual language models based on label-level supervision, characterized in that: The visual language model bidirectional preference optimization system includes: The dataset construction module is used to build a contrast generation strategy and a similarity filtering mechanism. Based on the contrast generation strategy and the similarity filtering mechanism, a contrast image is generated that maintains global semantic consistency with the original image while having visual feature differences. Based on the original image and the contrast image, a BiDPO dataset for bidirectional preference optimization is constructed. The loss function construction module is used to construct a bidirectional sequence-level loss function based on a bidirectional preference optimization strategy to align the image-answer pairs in the original direction and the reverse direction; to label the answers, a bidirectional label-level regularization loss function is constructed based on label-level rewards and regularization, so that the model increases labels with positive rewards and reduces labels with negative rewards; and a total loss function is constructed based on the bidirectional sequence-level loss function and the bidirectional label-level regularization loss function. The model training module is used to perform bidirectional training of the visual language model based on the BiDPO dataset by minimizing the total loss function, and to perform test calculations using the trained visual language model.

9. A device for bidirectional preference optimization of visual language models based on label-level supervision, characterized in that: The visual language model bidirectional preference optimization device includes a memory and a processor; the memory is used to store computer program code and transmit the computer program code to the processor; the processor is used to execute the visual language model bidirectional preference optimization method as described in any one of claims 1-7 according to the instructions in the computer program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that, when executed by a processor, implements the visual language model bidirectional preference optimization method as described in any one of claims 1-7.

Images