Human preference alignment method based on self-improvement large visual language model

Abstract

The invention discloses a human preference alignment method based on a self-improvement large visual language model. The method comprises the following steps: automatically constructing a preference data set; acquiring an image-question pair from the visual question and answer data set; applying various visual enhancements to the image, and driving a reference model to generate a group of candidate answers in combination with a question; the group of candidate answers are summarized and extracted into a more comprehensive'win 'text, and meanwhile, the text answers of the reference model to the original picture and the question are taken as'fall-fail' texts, so that preference data in an'image-inquiry-'win 'text-'fall-fail' text 'format are formed; secondly, based on the preference data set constructed in the first step, a direct preference optimization algorithm is applied to conduct alignment fine adjustment on a target visual language model, in the fine adjustment process, parameters of a visual encoder are kept frozen, and a low-rank adaptation layer (LoRA) is only introduced into an extended mode alignment module and a language decoder for training; carrying out iterative self-improvement on the model; after fine tuning is completed, taking the optimized model as a new reference model, and repeating the data construction process in the step 1 to generate preference data with higher quality for the next round of optimization; and the circulation is repeated, so that the continuous self-improvement of the model alignment capability is realized. According to the method, self-supervised preference alignment without manual annotation is realized, and a more comprehensive and high-quality'win 'text is generated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large-scale visual language models, and particularly relates to a large-scale visual language model human preference alignment method based on self-improvement. BACKGROUND

[0002] With the breakthrough of large language models (LLM), large visual language models (LVLM) have also developed rapidly and have shown excellent performance in tasks such as image-text question answering, image description, and visual reasoning. However, despite the powerful capabilities of these models, there are still major challenges in aligning their generated content with human preferences. Models sometimes generate text that does not match the visual input and human expectations, which can lead to the spread of misinformation and limit their application in the real world, especially in high-risk areas such as medical diagnosis.

[0003] In the medical diagnosis picture recognition scenario, doctors often need the model to give a professional, complete, and clinically logical explanation for X-ray films, CT images, or fundus images. However, existing LVLMs may have the following problems in such scenarios: (1) the output content is one-sided, only describing the surface features of the image, and lacks the combination of medical knowledge; (2) the answer has hallucinations, generating conclusions that do not match the actual image, or even giving misleading diagnoses; (3) the expression method does not conform to the medical terminology standard, and it is difficult to be adopted by the clinic.

[0004] Existing large visual language model human preference alignment methods can be mainly divided into three categories.

[0005] The first category is preference fine-tuning based on human annotation. This scheme first needs to collect a large-scale preference dataset and manually annotate it, and then use reinforcement learning or direct preference optimization (DPO) algorithms for fine-tuning. However, due to the time-consuming and expensive nature of human annotation, this method has high computational and financial costs, and may be difficult to cover all possible human preference distributions comprehensively.

[0006] The second category is preference generation based on external model scoring. To reduce the dependence on human annotation, some methods attempt to use powerful external models such as GPT-4 to generate preferences or perform scoring. However, this scheme also has its own challenges, such as inconsistent scoring mechanisms, discrete indicators that are difficult to accurately evaluate model capabilities, and high computational costs due to reliance on external models.

[0007] The third type is automatic generation of preference data based on text-side operations. Some recent studies attempt to create preference pairs by model self-evaluation or enhanced model output. For example, SEVA adopts a direct preference optimization algorithm for preference alignment, but it only relies on comparing the text output of the model for the original visual input and the enhanced visual input, limiting its ability to capture rich semantic connections. These methods generally have the drawback that they usually only focus on multiple operations on the text side while keeping the image side fixed when generating "winner" and "loser" texts. This text-only variant method ignores the multi-dimensional information contained in the visual content, which cannot be fully represented by the text description diversification variant, thus ignoring the key visual-text semantic interaction, and therefore cannot effectively capture the deep interaction between medical images and text.

[0008] For the key problem of aligning the output of large visual language models with human preferences, existing methods either rely on human annotation, which is costly, time-consuming, and has limited data diversity; or use external models to generate preferences, with inconsistent scoring mechanisms and lack of accuracy in evaluating model capabilities; or only operate on the text side, ignoring the mutual influence between images and text, indirectly causing a semantic gap between visual input and text output. SUMMARY

[0009] To address the shortcomings of existing large visual language model preference alignment methods, which rely on manual annotation, are costly, and fail to fully utilize visual-text interaction, the present invention proposes a self-supervised framework for a large visual language model human preference alignment method called SHAPE (Self-improved Holistic Alignment for Preference Enhancement).

[0010] The present invention aims to fully utilize the characteristics of large visual language models that are sensitive to visual disturbances, without the need for manual preference annotation, by automatically constructing high-quality competitive preference pairs, and iteratively improving the alignment capabilities of large visual language models.

[0011] The self-improved large visual language model human preference alignment method of the present invention includes the following steps: S1. Automatically construct a preference dataset; obtain "image-question" pairs from a visual question answering dataset; apply multiple visual enhancements to the image and combine the question to drive the reference model to generate a set of candidate answers; summarize and refine this set of candidate answers into a more comprehensive "winner" text, while the text answer of the reference model for the original picture and question is taken as the "loser" text, thereby forming a preference data in the format of "image-question-winner text-loser text"; S2. Based on the preference dataset constructed in step 1, the direct preference optimization algorithm is applied to fine-tune the alignment of the target visual language model. The target visual language model includes a visual encoder, a cross-modal alignment module, and a language decoder connected in sequence. The visual encoder is based on CLIP-ViT-L-336px and is used to encode the input image into high-dimensional visual features. The cross-modal alignment module is used to fuse visual and textual information. The language decoder is an autoregressive generative model used to generate the answer text. S3. Iterative self-improvement of the model; after fine-tuning, the optimized target visual language model is used as a new reference model, and the data construction process in step 1 is repeated to generate higher quality preference data for the next round of optimization; this process is repeated to achieve continuous self-improvement of the alignment capability of the target visual language model.

[0012] Preferably, in step S1, an initial preference dataset is constructed by automatically converting standard image-text supervision data pairs into (image, question, "winner" text, "loser" text) data that can be used for preference alignment. It includes the following steps: S11: From existing large-scale visual question answering datasets Image Acquisition in China - Problem Pair As input; the original image of the input. ,application A pre-defined visual enhancement method Generate a set An enhanced image The enhancement combination of contrast adjustment, diffusion model noise addition, and gamma correction is adopted to transform the image from three key dimensions: contrast, noise, and spatial region, respectively, to ensure the diversity of generated responses. S12: Transfer the original image Corresponding questions Input to reference model In the first iteration, For the initial target model Generate an initial text and define it as the "defeat" text. ; for each enhanced image generated in step 1-1 With the question Composition of Image-Problem Pair Input to the reference model In, generate a set One corresponding candidate text Due to the differences in the input images, this set of candidate texts typically exhibits diversity in detail and emphasis. S13: Construct a summarizing prompt. , the prompt word is concatenated with all the candidate responses generated in step 1-2 to form a long input text; the text is input into the reference model , the model will generate a single, more semantically rich, and more comprehensive summary text based on the comprehensive information of all candidate texts, which is defined as the "winning" text ; ; S14: combine the above elements into a complete preference data, the structure is ; store the data in the preference data set ; traverse all data in , repeat steps S11-S14 to finally form a complete preference data set for model fine-tuning.

[0013] Preferably, step S2 includes: using the preference data set constructed in step S1 , the target visual language model of the current round of iteration is fine-tuned by direct preference optimization algorithm, wherein represents the th round of iteration, and the steps of fine-tuning the target model are as follows: S21: using the preference data set constructed in step 1 , fine-tune the target model using the direct preference optimization algorithm; during fine-tuning, use partial parameter update, introduce low-rank adaptive layer (LoRA) for adjustment, keep the backbone parameters of visual encoder CLIP-ViT-L-336px frozen, and only update the trainable parameters in the cross-modal alignment module and the language decoder; introduce low-rank adaptive layer (LoRA) in the above modules, set its rank r to 1024, which is used to adjust the structural representation space of the model while keeping the parameter size controllable.

[0014] The goal of the direct preference optimization algorithm is to adjust the parameters of the target model to maximize the preference probability of the "winning" text while minimizing the preference probability of the "losing" text , while keeping the strategy consistent with the reference model ; the loss function of the direct preference optimization algorithm is as follows:

[0015] where is the Sigmoid function, is the input image-question pair; ​is a temperature coefficient hyper-parameter, used to control the weight of the reference model strategy, adjust the sensitivity of the loss function to the preferred reward, and the application sets value is 0.1; represents the expectation of all samples in the preferred data set , that is, the calculation is performed on the entire data set; S22: In the training process, the target model parameters that can minimize formula 1 are found by the AdamW optimization algorithm .

[0016] Preferably, the step of iterative self-improvement of the model in step S3 is as follows: S31: When the training of the first round is completed, the parameters of the target model are updated, in order to start the next round of self-improvement cycle, the current optimized target model needs to be used as the reference for the next round; therefore, the parameters of the reference model are updated to the parameters of the current target model ; S32: Steps S1 and S2 are repeatedly executed until a preset number of iterations or the model performance converges; in this "automatically generated data - optimized target model - updated reference model" cycle, the model uses its enhanced ability to construct better preferred data, thereby entering a positive, spiral self-improvement process, and continuously improving its alignment level.

[0017] Key points of the application: 1. Self-supervised preference alignment paradigm: the core is to propose a self-supervised learning framework that does not require manual annotation, but generates preference data by the model itself to complete the alignment.

[0018] 2. Generate diversified output using visual sensitivity: the key idea is to use the visual enhancement sensitivity of a large visual language model to the input image as an effective supervision signal to drive the model to generate diversified and semantically related output.

[0019] 3. Candidate text summary to build a more optimal overall sample: through a "self-summary" module, scattered information from multiple different visual dimensions is integrated into a comprehensive and coherent "winning" text to guide the model to learn to generate high-quality content that meets human preferences.

[0020] 4. Iterative self-improvement mechanism: through iterative cycles, the model continuously uses its enhanced ability to create higher quality training data, thereby achieving continuous, spiral improvement of capabilities.

[0021] The present application utilizes the sensitivity of large visual language models to visual enhancements, automatically constructs high-quality preference data pairs through self-summarization, and realizes effective alignment of large visual language models without any manual preference labeling, significantly improving the comprehensive performance and robustness of the model.

[0022] The present application has the following advantages: 1. Self-supervised preference alignment without manual annotation is achieved. Through an innovative data processing flow, the present application can automatically convert existing and rich visual question answering data into high-quality preference training data, reducing the dependence on expensive and time-consuming manual preference labeling, and significantly reducing the cost and threshold of large visual language model preference alignment.

[0023] 2. More comprehensive and high-quality "winning" texts are generated. By summarizing the outputs of multiple visual enhancement views, the "winning" texts generated by this method are superior to single-view outputs in terms of overallity, detail richness and semantic coherence. Experimental analysis shows that these "winning" texts have significantly lower perplexity, which means they are more stable and predictable learning targets, helping the model to fine-tune more robustly and efficiently.

[0024] 3. Full use of visual-text deep interaction. Unlike existing technologies that only operate on the text side, the present application fully utilizes the sensitivity of large visual language models to visual changes by introducing perturbations (i.e. image enhancement) on the visual side. This not only generates more diverse text outputs, but also maintains close semantic association between vision and text throughout the preference construction process, achieving a deeper alignment.

[0025] 4. Significantly improves model performance and efficiency. Extensive experiments on 12 benchmark tests demonstrate that the SHAPE framework of the present application achieves significant performance improvement on various model architectures and sizes. For example, on a 7B model, MMVet (comprehensive evaluation) score improved by 11.3%, and POPE (hallucination robustness) score improved by 8.0%. In addition, compared with traditional supervised fine-tuning (SFT), the present method can surpass the SFT performance using a large amount of labeled data (up to 132k) using only a small amount of unsupervised samples (16k), demonstrating extremely high data and training efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is a flowchart of the method of the present application.

[0027] Figure 2 is an example of single preference data construction of the present application. DETAILED DESCRIPTION

[0028] The technical solutions of the present application are further illustrated below with reference to the drawings.

[0029] The present embodiment provides a large visual model text generation method based on the self-improving large visual language model human preference alignment method of the present application, which can prevent the generation of text inconsistent with visual input and human expectations.

[0030] The present application provides a medical inquiry picture recognition method based on self-improving large visual language model human preference alignment, which comprises the following specific steps: Step S1: data acquisition and processing, and obtaining positive and negative samples.

[0031] In the preference data construction stage, starting from medical images and corresponding inquiry questions, various visual enhancements (such as contrast adjustment, gamma correction, noise addition, etc.) are applied to the original images to drive the reference model to generate diversified candidate answers. The original image output is used as the "defeated" text, while the multiple sets of answers generated for the enhanced images are summarized and integrated to form more complete and clinically meaningful "winning" texts, thereby obtaining the preference data pair with the structure of {image, question, winning text, defeated text} and storing it in the preference data set.

[0032] The preference modeling is based on the Bradley-Terry preference probability, and the preference probability of the following two candidate texts and can be written as:

[0033] wherein, is the internal reward function for the answer, is the winning text, is the defeated text.

[0034] Step S2: direct preference optimization fine-tuning based on the preference data set.

[0035] The present embodiment adopts a direct preference optimization (DPO) method to align the preferences of the target visual language model. The target model is composed of a visual encoder, a cross-modal alignment module, and a language decoder. The visual encoder adopts a CLIP-ViT-L-336px structure and is used to encode input medical images or other image tasks into high-dimensional visual features. The cross-modal alignment module is used to fuse the visual features and the semantic representation of the question. The language decoder is a self-recursive generation model used to generate answer texts.

[0036] ​In the fine-tuning process, instead of updating all parameters, a low-rank adaptive layer (LoRA) is introduced for adjustment, and the visual encoder remains frozen to avoid destroying the learned visual representation; a low-rank adaptive layer (LoRA) is introduced in the cross-modal alignment module and the language decoder, with the rank r set to 1024, and only the LoRA parameters and related trainable parts in these modules are optimized, thereby adjusting the model feature representation space without adding a large number of additional parameters. This structural fine-tuning method can improve the expression ability of the model under the driving of preference data, especially in terms of the desired expression style and information integrity of humans.

[0037] The preference data set constructed in step S1 is input into a direct preference optimization algorithm (DPO) to fine-tune the target visual language model, and the loss function is:

[0038] wherein, is a sigmoid function, is a temperature coefficient, and the present application sets to 0.1, and the optimal value range is determined through cross-validation. The optimization target gradually guides the model to generate answers closer to the doctor's expression style by maximizing the generation probability of "winning" texts and minimizing the generation probability of "losing" texts. In the medical consultation task, this means that the model can prefer to output explanations containing lesion sites, sizes, properties, and other diagnostic elements, rather than just staying in vague descriptions.

[0039] Step S3: Iterative self-improvement of the model.

[0040] After each round of fine-tuning is completed, the updated model is used as a new reference model, and steps S1 and S2 are re-executed to generate higher-quality preference data and continue optimization. As the iteration proceeds, the model can gradually generate outputs that conform to clinical logic when faced with more complex medical images, such as transitioning from "there is an abnormality" to "there is a 5mm nodule in the right upper lung with clear boundaries, suggesting a high likelihood of benignity." This spiral improvement mechanism ensures continuous improvement of the model in alignment with human preferences.

[0041] Step S4: Identify the medical consultation picture; To illustrate the practical application of the present application, the following takes lung CT images as an example: The patient uploads an image and raises a question "whether there is a lung nodule?". The reference model outputs "there is an abnormality" as the losing text for the original image; the candidate answers generated for multiple enhanced images include "a small nodule is seen in the right upper lung", "ground glass shadow is seen in the left lower lung", etc., and the winning text is generated after integration "a small nodule about 5mm is visible in the right upper lung, and ground glass shadow is visible in the left lower lung, further examination is recommended". After optimization by DPO training, the model is more inclined to output a conclusion similar to the winning text in subsequent reasoning, that is, it contains both clinical points and reasonable suggestions. The experimental results show that the present application has significant improvement in reducing hallucinations, improving medical terminology accuracy and diagnostic reference value.

[0042] The content described in the embodiments of the present specification is only a list of implementation forms of the inventive concept, and the protection scope of the present application should not be regarded as being limited to the specific forms stated in the embodiments, and the protection scope of the present application also extends to equivalent technical means that can be thought of by those skilled in the art according to the inventive concept.

Claims

1. A human preference alignment method based on a self-improving large-scale visual language model, comprising the following steps: S1. Automatically construct a preference dataset; Image-question pairs are obtained from a visual question-answering dataset. Various visual enhancements are applied to the images, and combined with the questions, the reference model is driven to generate a set of candidate answers. This set of candidate answers is summarized and refined into a more comprehensive "winning" text, while the textual answers of the reference model to the original image and question are used as "losing" text, thus forming preference data in the format of "image-question-'winning' text-'losing' text". S2. Based on the preference dataset constructed in step 1, the direct preference optimization algorithm is applied to fine-tune the alignment of the target visual language model. The target visual language model includes a visual encoder, a cross-modal alignment module, and a language decoder connected in sequence. The visual encoder is an image coding network based on CLIP-ViT-L-336px, which is used to encode the input image into high-dimensional visual features. The cross-modal alignment module is used to interactively fuse high-dimensional visual features with textual question representations; the language decoder is an autoregressive generative model used to generate answer text based on the fused representations; S3. Iterative self-improvement of the model: After fine-tuning, the optimized target visual language model is used as a new reference model, and the data construction process in step 1 is repeated to generate higher quality preference data for the next round of optimization; this process is repeated to achieve continuous self-improvement of the model alignment capability.

2. The method for aligning human preferences based on a self-improving large-scale visual language model as described in claim 1, characterized in that, In step S1, the initial preference dataset is constructed by automatically converting standard text-image supervision data pairs into data that can be used for preference alignment. It includes the following steps: S11: From existing large-scale visual question answering datasets Image Acquisition in China - Problem Pair As input; the original image of the input. ,application A pre-defined visual enhancement method Generate a set An enhanced image The enhancement combination of contrast adjustment, diffusion model noise addition, and gamma correction is adopted to transform the image from three key dimensions: contrast, noise, and spatial region, respectively, to ensure the diversity of generated responses. S12: Transfer the original image Corresponding questions Input to reference model In the first iteration, For the initial target model Generate an initial text and define it as the "defeat" text. ; Each enhanced image generated in step 1-1 With the question Composition of Image-Problem Pair Input to the reference model In, generate a set One corresponding candidate text Due to the differences in the input images, this set of candidate texts typically exhibits diversity in detail and emphasis. S13: Construct a summarizing prompt. Use the prompt "Please provide a comprehensive summary based on the following candidate answers" to highlight the key words. With all candidate responses generated in steps 1-2 The text is concatenated to form a long input text; this text is then input into the reference model. In the process, the model will generate a single, semantically richer, and more comprehensive summary text based on the combined information from all candidate texts. This text is defined as the "winning" text. ; S14: Combine the above elements into a complete preference data set, the structure of which is as follows: ; Store this data in a preference dataset. ; Traversal Repeat steps S11-S14 for all data to form a complete preference dataset for model fine-tuning.

3. The method for aligning human preferences based on a self-improving large-scale visual language model as described in claim 1, characterized in that, Step S2 includes: utilizing the preference dataset constructed in step S1 The target visual language model in the current iteration is optimized using the direct preference optimization algorithm. Perform preference optimization and fine-tuning, among which This indicates that this is the [number]th [number]. The steps for iterative fine-tuning the target model are as follows: S21: Using the preference dataset built in step 1 The direct preference optimization algorithm is used to fine-tune the target model. The fine-tuning process uses partial parameter updates and introduces a low-rank adaptation layer (LoRA) for adjustment. That is, the backbone parameters of the visual encoder CLIP-ViT-L-336px are frozen, and only the low-rank adaptation layer (LoRA) inserted in the cross-modal alignment module and the language decoder is trained and updated. The rank r of LoRA is set to 1024 to improve the model's ability to express the preference of "winning" text while ensuring that the parameter scale is controllable. The goal of the direct preference optimization algorithm is to adjust the target model. The parameters, so that it is comparable to the reference model While maintaining strategic consistency, maximize the impact on the "winning" text. The probability of preference, while minimizing the preference for "losing" text. The preference probability; the loss function of the direct preference optimization algorithm is shown below: in It is the Sigmoid function. It is an input image-question pair; It is a temperature coefficient hyperparameter used to control the weights of the reference model strategy and adjust the sensitivity of the loss function to preference rewards. This invention sets... The value is 0.1; Represents the preference dataset The expectation is calculated for all samples in the dataset, i.e., the expectation is calculated for the entire dataset. S22: During training, the AdamW optimization algorithm is used to find the target model parameters that minimize Equation 1. .

4. The method for aligning human preferences based on a self-improving large-scale visual language model as described in claim 1, characterized in that, The iterative self-improvement steps of the model in step S3 are as follows: S31: When the Round training completed, target model After the parameters are updated, in order to start the next round of self-improvement cycle, the current optimized target model needs to be used as the benchmark for the next round; therefore, the reference model will be used. The parameters are updated to the current target model. Parameters; S32: Repeat steps S1 and S2 until the preset number of iterations is reached. Or the target visual language model's performance converges; in this cycle of "automatically generating data - optimizing the target model - updating the reference model", the target visual language model uses its enhanced capabilities to construct better preference data, thus entering a positive, spiraling self-improvement process and continuously improving its alignment level.

Images