Knowledge-enhanced multi-modal large model entity-level preference alignment method

Abstract

The application discloses a kind of knowledge enhanced multimodal big model entity level preference alignment method, belong to artificial intelligence field.The application includes: collecting the data of text and image modal, text modal data is carried out fine-grained rewriting, the data of image modal extracts associated entity;Using the feature extraction network corresponding to each mode extracts feature, using a learnable alignment matrix will be adapted to the feature vector of other modal to the text feature vector;The most similar feature is retrieved in knowledge base and is spliced to obtain the preferred feature by big model, and the feature with the maximum semantic difference is spliced to obtain the unpreferred feature by big model;The preferred feature, original feature and unpreferred feature are used as the input of target task big model for training.The application seamlessly integrates text and visual modal, retrieves associated features from multimodal knowledge graph for preference alignment to bridge the preference gap between model and human, effectively improves the ability of multimodal big model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more particularly to a knowledge-enhanced multimodal large model entity-level preference alignment method. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, especially the advancements in large-scale modeling and multimodal learning, more and more industries are beginning to apply these technologies to solve practical problems. In reality, AI has demonstrated enormous potential, particularly in intelligent question-answering services. To achieve these intelligent services, it is typically necessary to process data from multiple modalities, including text, images, and videos. Although existing multimodal learning methods have made some progress, in areas with human preferences, such as cultural tourism route recommendations and personalized services, efficiently integrating and optimizing this data still faces many challenges.

[0003] First, traditional services in the field of human preferences typically rely on single-modal data. For example, many existing travel recommendation systems primarily rely on textual data such as users' browsing history and reviews. However, single-modal data is often insufficient to fully capture users' interests and preferences. In practice, tourists' interests are not only reflected in textual descriptions but may also be presented through images, videos, and other forms of expression. Therefore, how to effectively integrate data from different modalities to more comprehensively understand user needs is a crucial issue in the field of human preferences. Second, while existing multimodal learning methods have achieved some success in data fusion, they are usually limited to simple feature-level fusion. A more complex problem lies in how to accurately align these modalities so that the features of each modality can be effectively combined within the same semantic space. For example, in the cultural tourism industry, data from different sources (such as user reviews on social media, images and text reviews from travel websites, etc.) often have different formats and qualities. How to effectively process and align these heterogeneous data is a significant challenge.

[0004] Furthermore, most existing technologies neglect the issue of human preferences. For example, in the cultural tourism sector, user needs and interests are diverse and personalized. Each user's preferences may not only be related to their past behavioral records but also closely linked to multiple factors such as cultural background and travel goals. The core issue of preference alignment lies in how to accurately identify and capture users' personalized preferences through models, and on this basis, provide them with tailored recommendations and services. For example, some users may prefer historical and cultural attractions, while others may prefer natural scenery tourism. This difference in personalized needs requires models to dynamically adjust and optimize recommendation strategies, thereby improving user experience and ensuring that recommendations better match users' actual needs, rather than simply being based on historical data or behavioral patterns. Summary of the Invention

[0005] This invention proposes a knowledge-enhanced multimodal large model entity-level preference alignment method, which aims to realize question-answering functionality based on multimodal large models. It seamlessly integrates text and visual modalities, retrieves related features from existing multimodal knowledge graphs for preference alignment, and bridges the preference gap between the target task model and humans, effectively improving the capabilities of multimodal large models.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A knowledge-enhanced multimodal large model entity-level preference alignment method includes the following steps:

[0008] Step 1: Rewrite the text modality data using question-answer pairs, and extract entity-level images from the image modality data that are associated with the rewritten text data. Based on the extracted images and the rewritten text data, construct strongly correlated text-image pairs.

[0009] Step 2: Perform feature encoding on each strongly correlated text-image pair to obtain the text embedding features H of the question in the text data of the text-image pair. t And the visual features X of the image in the text-image pair. v ;

[0010] Text embedding features H t With visual feature X v Feature alignment is performed to obtain encoded and aligned strongly correlated text-image pairs;

[0011] Step 3: In the selected multimodal knowledge graph related to the target task, retrieve the entity features and their proximity relationships associated with the text-image pairs obtained in Step 2 based on similarity.

[0012] The similarity retrieval results are concatenated and then input into the selected large model to construct the answer to each question;

[0013] Step 4: Encode the answer to obtain preference features and non-preference features. Based on the preference features, non-preference features and original features of each question, perform preference alignment optimization training on the target large model.

[0014] Furthermore, in step 1, the rewriting of the text modality data through question-and-answer pairs specifically includes:

[0015] Break down the answers in a question-and-answer pair into individual facts as atomic knowledge;

[0016] Extract the atomic knowledge set K and the answer a:

[0017]

[0018] Among them, K i Let q represent the i-th question. i The set of atomic knowledge, Extract() represents the extraction of atomic knowledge from the large language model through instruction prompts, a i Let q represent the i-th question. i The corresponding answer, This indicates that the answer is a. i The extracted knowledge of the j-th atom, |K i | represents the set of atomic knowledge K i The number of atomic knowledge items included;

[0019] Question q i Rewritten with atomic knowledge K i Related fine-grained issues:

[0020] q i =Rewrite(q i ,K i ),

[0021] Here, Rewrite(,) represents rewriting the large language model's question q via instruction prompts. i .

[0022] Furthermore, in step 2, the visual features X are aligned using a learnable alignment matrix. v Embedding features H into text t Adapt to align features.

[0023] Furthermore, in step 2, a Transformer-based text encoder is used to obtain the text embedding features H. t Visual features X are extracted using a pre-trained CLIP (Contrastive Language-Image PreTraining) encoder. v .

[0024] Furthermore, in step 2, feature alignment specifically involves:

[0025] Based on the adapter, visual feature X v Converted into text embeddings H that can be understood by large language models v =W v ·X v +b v Among them, W v b v These represent the learnable adapter matrix and adapter bias vector for text and visual feature alignment, respectively.

[0026] Acquiring aligned visual features H using single-head attentionv ′:

[0027]

[0028] Where, d k H represents the text embedding feature t Dimensions.

[0029] Furthermore, in step 3, cosine similarity is used for similarity retrieval.

[0030] Furthermore, in step 4, the most similar entity features retrieved are concatenated and then processed through a large language model to obtain preferred features, while the entity features with the greatest semantic differences retrieved are concatenated and then processed through a large language model to obtain unpreferred features.

[0031] Furthermore, in step 4, the knowledge k1 retrieved from the entity with the highest similarity in the graph through one-hop retrieval is used as the background knowledge of the current problem and input into the large language model to obtain the preference feature; and the background knowledge retrieved through two-hop retrieval is input into the large language model to obtain the non-preference feature.

[0032] Furthermore, in step 4, when performing preference alignment optimization training on the target large model, the training objective function is set as follows:

[0033]

[0034] in, This indicates the fine-tuned training loss used. Let λ represent the alignment target loss, and λ represent the preset weights.

[0035] Among them, fine-tuning training loss and alignment target loss They are respectively:

[0036]

[0037] Where σ() is the sigmoid function, S i S represents the preference score for the i-th answer. j Indicates less than S i Preference scores for all answers Indicates the use of preference pairs (S) i ,S j ) count, adaptive weights S min S represents the lowest preference score among all answers. max This represents the highest preference score among all answers; This represents the mean log-likelihood. q represents the predicted probability of the target large model M. i Let a represent the i-th question.i q i The corresponding answer, a i,j Indicates a i The j-th token, a i,＜j Indicates a i All tokens preceding the j-th token, where I represents the hint template.

[0038] The technical solution provided by this invention brings at least the following beneficial effects:

[0039] This invention, by introducing retrieval from an external multimodal knowledge base (i.e., a selected multimodal knowledge graph relevant to the target task) and optimizing the alignment of multimodal features, not only enables effective data fusion between different modalities but also bridges the preference gap between the model and the user through the alignment optimization of preference features. Through deep learning and optimization of user interests and preferences, this invention improves the personalization and intelligence of recommendations based on human preferences, better meeting the personalized needs of different users. Attached Figure Description

[0040] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0041] Figure 1 A schematic diagram illustrating the processing steps of a knowledge-enhanced multimodal large model entity-level preference alignment method provided in an embodiment of the present invention;

[0042] Figure 2 This is a schematic diagram of a specific training method based on a multimodal large model. Detailed Implementation

[0043] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions of the embodiments of the present invention will be described in detail and completely below with reference to the accompanying drawings. Obviously, the embodiments described with reference to the accompanying drawings are exemplary and intended to explain the present invention, but should not be construed as limiting the present invention.

[0044] This invention provides a knowledge-enhanced multimodal large model entity-level preference alignment method, applicable to the intersection of multimodal learning and AI-powered intelligent question-answering services related to human preferences. Specifically, the method proposed in this invention involves using multimodal data (including text, images, etc.) to train and optimize a multimodal large model to achieve a more intelligent question-answering system.

[0045] In one embodiment, such as Figure 1As shown in the embodiment of the present invention, a knowledge-enhanced multimodal large-scale model entity-level preference alignment method includes: First, collecting data from text and image modalities, performing fine-grained rewriting of the text modal data, and extracting associated entities from the image modal data; second, using feature extraction networks corresponding to each modality to extract features, and using a learnable alignment matrix to adapt the feature vectors of other modalities to the text feature vectors; retrieving the most similar features from the selected external knowledge base (i.e., the selected existing multimodal knowledge graph), concatenating them, and obtaining preferred features through a large language model, and concatenating features based on the maximum semantic difference, and obtaining unpreferred features through a large language model; using the preferred features, the original features, and the unpreferred features together as input to the target task's large model (multimodal cultural tourism large model) for training. This invention realizes question-answering and other functions based on a multimodal large model, seamlessly integrating text and visual modalities, retrieving associated features from the multimodal knowledge graph for preference alignment to bridge the preference gap between the model and humans, and effectively improving the capabilities of the multimodal large model.

[0046] In one embodiment, such as Figure 2 The specific implementation steps of the knowledge-enhanced multimodal large model entity-level preference alignment method provided in this embodiment include:

[0047] Step S1: Perform fine-grained rewriting of question-answer pairs on the text modality data, and extract entity-level images that are associated with the rewritten text from the image modality data as strongly correlated text-image pairs.

[0048] In this embodiment, the QA pairs are rewritten for LLM fine-grained knowledge. The answer is broken down into individual facts as atomic knowledge. Atomic knowledge is a concise statement expressing a fact; it is a more granular unit than a sentence. Specifically, the atomic knowledge set K and the answer a are extracted using the following formula.

[0049]

[0050] Among them, Extract(*) extracts atomic knowledge by prompting the OpenAI model with instructions, a i This represents the answer to the i-th question. This indicates that the answer (a) i The knowledge of the j-th atom extracted.

[0051] Then, the question q i Rewritten with atomic knowledge K i Related fine-grained issues.

[0052] q i =Rewrite(q i ,K i ),

[0053] Among them, Rewrite(*) prompts the OpenAI model to implement the rewriting problem, K i This represents the set of atomic knowledge corresponding to the i-th question obtained by Extract(*).

[0054] Finally, LLM is used to assist in the extraction of in-image entities associated with the question. Specifically, an LLM-RA model and context learning are used to facilitate the generation of structured entities and their attributes, with the instruction from the prompting model: "Given an image, a question, and an answer description, output entities of the image classes that the question might be interested in." LLM is then instructed to generate entity-level images associated with the rewritten text, resulting in strongly correlated text-image pairs.

[0055] Step S2: Encode multimodal data (such as text and images) to extract features, and align strongly correlated text and image data. The result contains encoded and aligned strongly correlated text-image pairs.

[0056] For question Q, a Transformer-based text encoder is used to obtain the text embedding features H. t .

[0057] To eliminate cross-entity interference during inference, the extracted entity-level images are independently encoded. For example, a pre-trained CLIP encoder can be used to extract the visual features X of the entity-level images. v To enable LLM to understand multimodal knowledge node embeddings, this embodiment introduces an adapter module to integrate X... v Convert to text embeddings that LLM can understand v This module aims to bridge the inherent gap between multimodal knowledge and text, thereby achieving a more seamless reconciliation. The module formula is shown below:

[0058] H v =W v ·X v +b v

[0059] Among them, W v This represents a learnable adapter matrix for text and visual feature alignment, b v This represents a learnable bias vector.

[0060] Then, using single-head attention, the final visual features H associated with the text embedding features are obtained through the following function. v ′.

[0061]

[0062] Where, d k Text feature Ht The dimension, H t This represents text embedding features.

[0063] Step S3: Retrieve the associated entity features and their proximity relationships of text image pairs in the multimodal knowledge graph based on similarity, and then construct the answer by concatenating the retrieval results and using the OpenAI model.

[0064] Background knowledge is extracted based on similarity retrieval using each encoded and aligned entity-level image and QA pair, where the similarity is based on the cosine similarity between the two, as shown in the following formula.

[0065] sim(i,j)=Cosine(H(q i ),H(g j ))

[0066] Where H represents the searcher, q i Let g represent the i-th question. j This represents the j-th piece of knowledge in the graph after being encoded by the graph neural network RGAT.

[0067] For each question, the knowledge k1 retrieved from the entity with the highest similarity in the graph through a one-hop is used as the background knowledge of the current question and input into the OpenAI model to obtain the result a1; the answer corresponding to the original question is used as a2, which has no background knowledge, i.e., k2 is an empty set; the background knowledge retrieved through two hops is used as k3 and input into the OpenAI model to obtain the result a3.

[0068] Through the above operations, we obtain the answer A = {a1, a2, a3} for the same question q using a large model.

[0069] Step S4: Construct preference pairs based on the order of the answers for the multimodal cultural tourism big data model to perform entity-level preference alignment optimization training.

[0070] The best answer, with the highest preference score, is obtained by using the most relevant one-hop knowledge retrieved from the multimodal knowledge graph through a large model. The second best answer is obtained without any background knowledge. However, based on experience, background knowledge retrieved in the graph after two or more hops often misleads the model and creates illusions, resulting in the lowest preference score. Therefore, preference pairs can be constructed based on this. First, a preference score is calculated for each answer, as shown in the formula below.

[0071]

[0072] in, This represents the mean log-likelihood. a represents the predicted probability of model M. i,j Indicates a iThe j-th token, a i,＜j Indicates a i All tokens preceding the j-th token.

[0073] This formula represents the average log-likelihood of each answer. A higher score indicates that the model believes the current answer is more likely to occur.

[0074] To ensure that the model's preferences align with the aforementioned preferences, the following alignment objective was designed.

[0075]

[0076] Where σ is the sigmoid function, S i S represents the score corresponding to the i-th answer. j Indicates less than S i The scores corresponding to all answers, This represents the count used for this preference pair.

[0077] Since different answers have different text quality and preference levels, an adaptive weight can be used to control the influence of different preferred answers, that is, the alignment target is changed to the following formula.

[0078]

[0079] The adaptive weight μ i The calculation is performed using the following formula.

[0080]

[0081] Where S i S represents the score corresponding to the i-th answer. min S represents the lowest score among all answers. max This represents the highest score among all answers.

[0082] The fine-tuning training formula combining VFT (vanilla fine-tuning) is as follows, and the formula for VFT is shown below.

[0083]

[0084] in, This represents the mean log-likelihood. q represents the predicted probability of model M. i Let a represent the i-th question. i,j Indicates a i The j-th token, a i,＜j Indicates a i All tokens preceding the j-th token, where I represents the hint template.

[0085] Combining the two aligned objective functions, and adding λ to represent the training hyperparameters, we obtain the final overall training objective function as follows:

[0086]

[0087] Through the above description of the implementation methods, those skilled in the art can clearly understand the specific technical details of each step. It should be noted that the above methods are only used to illustrate the technical solution of the present invention, and not to limit it. Those skilled in the art can still make equivalent substitutions for some technical features, such as using VIT to extract features in the feature extraction stage of step S2, and obtaining the answer through various different open-source large models in step S3.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the application, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A knowledge-enhanced multimodal large model entity-level preference alignment method, characterized in that, Includes the following steps: Step 1: Rewrite the text modality data using question-answer pairs, and extract entity-level images from the image modality data that are associated with the rewritten text data. Based on the extracted images and the rewritten text data, construct strongly correlated text-image pairs. Step 2: Perform feature encoding on each strongly correlated text-image pair to obtain the text embedding features of the question in the text data of the text-image pair. and the visual features of images in text-image pairs. ; Text embedding features With visual features Feature alignment is performed to obtain encoded and aligned strongly correlated text-image pairs; Specifically, a Transformer-based text encoder is used to obtain text embedding features. Visual features are extracted using a pre-trained CLIP encoder. ; Step 3: In the selected multimodal knowledge graph related to the target task, retrieve the entity features and their proximity relationships associated with the text-image pairs obtained in Step 2 based on similarity. The similarity retrieval results are concatenated and then input into the selected large model to construct the answer to each question; Step 4: Encode the answer to obtain preference features and non-preference features. Based on the preference features, non-preference features and original features of each question, perform preference alignment optimization training on the target large model.

2. The method as described in claim 1, characterized in that, Step 1, rewriting the text modality data using question-and-answer pairs, specifically includes: Break down the answers in a question-and-answer pair into individual facts as atomic knowledge; Extract the atomic knowledge set K and the answer a: ； in, Represents the i-th question A collection of atomic knowledge, This indicates that atomic knowledge is extracted from the large language model through instruction prompts. Represents the i-th question The corresponding answer, Indicates about the answer The extracted knowledge of the j-th atom, Represents a set of atomic knowledge The number of atomic knowledge items included; The problem Rewritten with atomic knowledge Related fine-grained issues: ， in, This indicates the issue of rewriting large language models through instruction prompts. .

3. The method as described in claim 1, characterized in that, In step 2, visual features are aligned using a learnable alignment matrix. Embedding features into text Adapt to align features.

4. The method as described in claim 1, characterized in that, In step 2, feature alignment specifically involves: Based on the adapter, visual features Converted into text embeddings that can be understood by large language models ,in, , These represent the learnable adapter matrix and adapter bias vector for text and visual feature alignment, respectively. Acquiring aligned visual features using single-head attention : ； in, Representing text embedding features Dimensions.

5. The method as described in claim 1, characterized in that, In step 3, cosine similarity is used for similarity retrieval.

6. The method as described in claim 1, characterized in that, In step 4, the most similar entity features retrieved are concatenated and then processed through a large language model to obtain the preferred features, while the entity features with the greatest semantic differences retrieved are concatenated and then processed through a large language model to obtain the unpreferred features.

7. The method as described in claim 1, characterized in that, In step 4, the entities with the highest similarity in the retrieved graph are used to retrieve knowledge through a one-hop search. As background knowledge for the current problem, it is input into a large language model to obtain preference features; And the background knowledge retrieved through two hops is used as input into the large language model to obtain the unbiased features.

8. The method as described in claim 1, characterized in that, In step 4, when performing preference alignment optimization training on the target large model, the training objective function is set as follows: ； in, This indicates the fine-tuned training loss used. Indicates the alignment target loss. Indicates the preset weights; Among them, fine-tuning training loss and alignment target loss They are respectively: ； ； Among them, For the sigmoid function, This represents the preference score for the i-th answer. Indicates less than Preference scores for all answers Indicates the use of preference pairs The count, adaptive weights , This represents the lowest preference score among all answers. This represents the highest preference score among all answers; This represents the mean log-likelihood. This represents the predicted probability of the target large model M. This represents the i-th question. express The corresponding answer, express The j-th token, express All tokens preceding the j-th token, where I represents the hint template.

Images