Visual feature generation method for improving visual language model combinatorial reasoning

Abstract

The invention provides a visual feature generation method for improving visual language model combinatorial reasoning, and belongs to the technical field of cross-modal semantic enhancement and image feature learning, and the method comprises the steps: constructing a data set, and obtaining a semantic offset between a positive description and a negative description to generate a difference vector; generating visual disturbance through the difference vector, guiding image feature modification, and obtaining visual features which keep semantic consistency with negative description of the text; and the generated visual negative features are applied to contrast learning of a visual-language model, and the performance of the model in a combined reasoning task is optimized. The method has the advantages that disturbance vectors for visual features can be generated through semantic offset guidance of positive and negative samples of a text, complementary information between the text and an image is fully mined, and the quality of generation of the visual negative features is improved; complex semantic information is processed more effectively in the evaluation process, and finally the ability of the model in processing the combinatorial reasoning task is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of cross-modal semantic enhancement and image feature learning, and specifically relates to a visual feature generation method for improving combined reasoning of a visual language model. Background Art

[0002] In recent years, multimodal learning has made significant progress in the fields of computer vision, natural language processing, and artificial intelligence, and has gradually become a hot research topic. Among them, vision-language models (VLMs) are an important research object of multimodal learning. Due to their ability to model the association between images and texts, they have received widespread attention and application. By combining visual and text information, these models have achieved excellent performance in tasks such as image classification, object detection, and image description generation. However, although these models perform well on many tasks, they still face many challenges when faced with complex semantic reasoning tasks. These challenges mainly stem from the model's shortcomings in fine-grained semantic reasoning, negative sample generation strategies, and cross-modal feature alignment.

[0003] Compositional reasoning tasks require models to be able to analyze and understand the properties of multiple objects and the relationships between objects in complex scenes. In compositional reasoning tasks, models not only need to handle simple matches between images and texts, but must also be able to effectively capture fine-grained semantic combinations. For example, Krishna et al. proposed the Visual Genome dataset in the paper "Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations", which aims to help training models better capture the complex interactions of objects and relationships in scenes by combining visual and textual information, thereby supporting compositional reasoning tasks. However, existing visual-language models often perform poorly in handling compositional reasoning tasks, and have difficulty accurately identifying and distinguishing subtle differences between different objects, relationships, and their attributes. For example, Johnson et al. pointed out in the paper "CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning" that current models usually lack sufficient reasoning capabilities when understanding compositional relationships and semantic changes between objects, especially when faced with semantically complex scenes, and perform less well than expected. Therefore, how to make the model effectively cope with the fine-grained semantic understanding in combinatorial reasoning tasks is an urgent problem to be solved in the research of vision-language models.

[0004] Negative sample generation is one of the important means to improve the generalization ability in model training. By using the generated negative samples for training, the model can effectively learn the ability to distinguish between positive and negative samples, thereby improving the ability to distinguish fine-grained semantic differences. However, existing negative sample generation methods often focus too much on the generation of text negative samples and ignore the generation and application of image negative samples. For example, Doveh et al. proposed a data augmentation method by generating structured text in the paper "Teaching Structured Vision & Language Concepts to Vision & Language Models", and used the generated positive and negative structured text samples to fine-tune the visual-language model. However, this method focuses on the generation of positive and negative samples at the text level without considering negative samples at the image level, which will lead to insufficient generalization and robustness of the model in distinguishing visual details in the image. Moreover, the existing image negative sample generation methods mainly rely on two strategies: one is to modify the image according to the semantic changes of the text through image restoration tools, and the other is to sample by using the data similarity sampling method to match the image in the dataset that is most similar to the negative description of the text. However, although the image inpainting tool-based method can generate negative samples of images that match the text description, the generated images often have visual inconsistencies or semantic ambiguity. When the inpainting tool modifies certain details in the image (such as color, position, etc.) according to the changes in the text, the modified image may be visually unnatural and even destroy the overall semantic consistency of the scene. For example, when modifying "blue car" to "red car", the image inpainting tool may not be able to accurately handle the color change, resulting in unnatural image colors or distorted details. This unreasonable visual perturbation will directly affect the quality of negative samples and interfere with the training of the model. Although the sampling method based on data similarity can quickly match image negative samples, since this sampling method usually relies on a limited range of data sets, it is difficult to match images that match the negative description. Such low-quality image negative samples will further affect the performance of the model. Summary of the invention

[0005] The technical problem to be solved by the present invention is that when the repair tool modifies certain details in the image (such as color, position, etc.) according to the changes in the text, the modified image is visually unnatural, and the image repair tool may not be able to accurately handle the color changes, resulting in unnatural image colors or distorted details. In view of the shortcomings of the prior art, a visual feature generation method for improving the combined reasoning of visual language models is provided, which can solve the above problems.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: including constructing a data set, wherein the data is an image-text pair consisting of an image and its text description; generating a negative description of the text by exchanging the position or attribute of the entity; obtaining a semantic offset between the positive description and the negative description to generate a difference vector through the generated negative description of the text; generating a visual perturbation through the difference vector to guide the modification of image features and obtain visual features that maintain semantic consistency with the negative description of the text; applying the generated visual negative features to the contrastive learning of the visual-language model to optimize the performance of the model in the combinatorial reasoning task.

[0007] Further, a method for constructing a data set, wherein the data is an image-text pair consisting of an image and its text description, comprises: constructing a data set, wherein the data is an image-text pair consisting of an image and its text description, and converting the text T in the data set into orig Parsed into a scene graph representation, for the text T orig Parsing obtains triples (O1, R, O2) or attribute pairs (A1, O1) and (A2, O2); where O1 and O2 represent the text T orig Different entities in, A1 and A2 represent the attribute information corresponding to the entity.

[0008] Further, by exchanging the position or attribute of the entity, the method of generating a negative description of the text includes:

[0009] Assume that after parsing, a triple (O1, R, O2) is obtained, and the positions of entities O1 and O2 are swapped in the triple (O1, R, O2) by Swap((O1, R, O2)) = (O2, R, O1);

[0010] Assume that after parsing, we get attribute pairs (A1, O1) and (A2, O2), and attribute pairs (A1, O1) and (A2, O2) exchange attribute information through Swap((A1, O1), (A2, O2)). The calculation method is:

[0011] Swap represents a swap operation.

[0012] Furthermore, the method of obtaining the semantic offset between the positive description and the negative description and generating a difference vector by generating a negative description of the text includes: using a pre-trained visual-language model to generate a difference vector between the positive description and the negative description. orig and T neg Perform embedding processing to generate semantic embedding vectors of text and the semantic embedding vector of the negative description of the text Get and The semantic shift difference vector Δ e , capturing the semantic changes of text, calculation method: Among them, Δ e Represents the semantic shift from positive samples to negative samples.

[0013] Furthermore, the method of generating visual disturbance through the difference vector, guiding image feature modification, and obtaining visual features that maintain semantic consistency with the negative description of the text includes:

[0014] From the image I orig Extract image features Generate negative descriptions with text T neg Corresponding visual negative features The calculated difference vector Δ e With image features Combined to generate visual negative features Calculation method:

[0015]

[0016] An optimization system for improving the performance of visual language model combination reasoning tasks, used to execute the above-mentioned visual feature generation method, characterized by comprising:

[0017] A data set building module is used to build a data set for generating negative samples;

[0018] A visual negative feature generation module for extracting semantic embeddings of text and negative descriptions using a pre-trained vision-language model;

[0019] Model fine-tuning module, used to use the generated visual negative features, text features of the image, and image features for fine-tuning the vision-language model

[0020] An embedded vision-language reasoning system, characterized by comprising an optimization system for improving the performance of a vision-language model combination reasoning task, and further comprising:

[0021] A data acquisition module is used to acquire images from a camera or sensor and generate corresponding real-time text descriptions;

[0022] The real-time reasoning and feedback module is used to perform reasoning based on the generated visual negative features and provide real-time feedback to the user or system based on the reasoning results.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] Compared with existing methods, it can generate perturbation vectors for visual features through the semantic offset of positive and negative text samples, fully tap the complementary information between text and image, and improve the quality of visual negative feature generation. The visual language model can more effectively learn the subtle differences between positive and negative samples by using high-quality visual negative features for contrastive learning, and thus more effectively process complex semantic information during the evaluation process, ultimately improving the model's ability to handle combined reasoning tasks.

[0025] The reason why the present invention has the above-mentioned effect is that the difference vector generated by semantic offset calculation effectively guides the disturbance at the image level, and realizes the semantic and visual alignment of text and image. The visual negative features generated by the method described in the present invention can effectively enhance the model's ability to recognize fine-grained semantic differences, thereby improving the generalization ability in combined reasoning tasks.

[0026] The present invention can be used in combination with existing data enhancement technologies, such as text expansion technology or image restoration technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0028] Figure 1 : Schematic diagram of the technical process in Example 1 of the present invention;

[0029] Figure 2 : A structural diagram of the combined reasoning process in Example 1 of the present invention. DETAILED DESCRIPTION

[0030] In order to better understand the present invention, the content of the present invention is further clearly described below in conjunction with the embodiments and the accompanying drawings, but the protection content of the present invention is not limited to the following embodiments. In the following description, a large number of specific details are provided to provide a more thorough understanding of the present invention. However, it is obvious to those skilled in the art that the present invention can be implemented without one or more of these details.

[0031] Example 1: See Figure 1 , a visual feature generation method for improving the combined reasoning of a visual language model in this embodiment, comprising:

[0032] (1) Construct a dataset, where the data is an image-text pair consisting of an image and its text description. orig Process the text T orig Transformed into a scene graph representation, that is, for each text T origWe get the triples (O1, R, O2) and attribute pairs (A1, O1) and (A2, O2). O1 and O2 represent different entities in the same sentence, and A1 and A2 represent the attribute information of these entities. By exchanging the positions or attributes of the entities, we can generate a negative description of the text. neg Specifically, for the triple (O1, R, O2), the positions of entities O1 and O2 are swapped by Swap((O1, R, O2)) = (O2, R, O1); for the attribute pairs (A1, O1) and (A2, O2), the attribute information is swapped by Swap((A1, O1), (A2, O2)). The calculation method is:

[0033]

[0034] Among them, Swap represents a swap operation.

[0035] In order to capture the semantic shift from negative text samples to positive text samples, a pre-trained visual-language model is used to train the text T orig and T neg Perform embedding processing to generate their semantic embedding vectors and calculate and The semantic difference vector Δ e , calculation method:

[0036] Among them, Δ e Represents the semantic shift from positive samples to negative samples, which is used to capture the semantic changes of text.

[0037] (2) In order to orig Extract image features Generate negative descriptions with text T neg The corresponding visual negative features are calculated by the difference vector Δ e With image features Combined to generate visual negative features Calculation method:

[0038] The visual negative features generated by the above method can not only provide the model with visual information that is semantically consistent with the negative description of the text, but also help the visual-language model effectively learn the semantic associations between cross-modal data. The generated visual negative features are used for multimodal contrastive learning, and the tasks of multimodal contrastive learning include but are not limited to combinatorial reasoning.

[0039] The following experiment uses the Visual Genome (VG) dataset to test the effect of this embodiment. The dataset was proposed by the paper "Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations" (authors: Ranjay Krishna, Yuke Zhu, Oliver Groth, and Michael Bernstein), published in 2017 (International Journal of Computer Vision), and contains rich images and their corresponding text descriptions and relationship annotations. The VG dataset contains 108,077 images and their fine-grained annotations, including objects, attributes, and relationship information. This dataset requires the model to identify the corresponding text descriptions in multiple text descriptions based on image data. We tested the following three methods as experimental comparisons:

[0040] Existing method 1: The paper "Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Mining" proposes a negative sample mining technology method, which generates text negative samples for fine-tuning the visual-language model by replacing objects or backgrounds in images and generating corresponding slight modifications in text, thereby improving its performance in combinatorial reasoning tasks.

[0041] Existing method 2: The paper "When and Why Vision-Language Models Behave Like Bags-Of-Words, and What to Do About It" proposes a method for generating negative samples by shuffling the word order of the text. It also introduces the Attribution, Relation, and Order (ARO) benchmark to systematically evaluate the model's ability to understand attributes, relations, and word order.

[0042] Existing Method 3: The paper “Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Models” proposes a method to improve the quality of text in image-text datasets. It fine-tunes the visual-language model by generating high-quality image descriptions, thereby improving the model’s performance in processing compositional reasoning tasks.

[0043] The present invention: the method of this embodiment.

[0044] The experiment uses the accuracy metric to evaluate the performance of the model in the combinatorial reasoning task. Accuracy refers to the ratio of correct reasoning results on the test set. The higher the accuracy, the better the reasoning effect.

[0045] Table 1. Experimental results of the present invention are shown

[0046]

[0047] As can be seen from Table 1, the proposed method is superior to the existing methods in terms of accuracy. The existing method 1 has poor quality in generating negative samples, which makes it difficult for the model to effectively learn high-quality negative representations, thus limiting the model performance; and the existing methods 2 and 3, although they perform well, still have limitations in capturing fine-grained semantic differences. The proposed method generates visual negative features through higher-quality visual features to fine-tune the visual-language model, effectively improving the performance of the model in processing combined reasoning tasks.

[0048] Beneficial effects:

[0049] 1. Compared with existing methods, it can generate perturbation vectors for visual features through the semantic offset of positive and negative text samples, fully tap the complementary information between text and image, and improve the quality of visual negative feature generation. The visual language model can more effectively learn the subtle differences between positive and negative samples by using high-quality visual negative features for contrastive learning, and thus more effectively process complex semantic information during the evaluation process, ultimately improving the model's ability to handle combined reasoning tasks.

[0050] The reason why the present invention has the above-mentioned effect is that the difference vector generated by semantic offset calculation effectively guides the disturbance at the image level, and realizes the semantic and visual alignment of text and image. The visual negative features generated by the method described in the present invention can effectively enhance the model's ability to recognize fine-grained semantic differences, thereby improving the generalization ability in combined reasoning tasks.

[0051] The present invention can be used in combination with existing data enhancement technologies, such as text expansion technology or image restoration technology.

[0052] Embodiment 2: An optimization system for improving the performance of visual language model combination reasoning tasks, used to execute the visual feature generation method for improving visual language model combination reasoning described in Embodiment 1, comprising: a data set establishment module, responsible for establishing a data set for negative sample generation. In this embodiment, the constructed data set contains a rich set of image and text pairs. By constructing a scene graph from the text of the data set, and perturbing the object attributes and relationships in the image based on exchanging the entity or attribute positions of the scene graph nodes, a negative description of the text with contrast is generated; specifically, the data set establishment module is used to execute step (1) of Embodiment 1;

[0053] The visual negative feature generation module is responsible for extracting the semantic embedding of the text and the negative description using the pre-trained visual-language model. The semantic difference vector is generated by calculating the semantic offset between the text and the negative sample text. The difference vector is combined with the image features to generate the visual negative feature corresponding to the negative description of the text; specifically, the visual negative feature generation module is used to perform step (2) of Example 1;

[0054] The model fine-tuning module is responsible for using the generated visual negative features, image text features, and image features to fine-tune the visual-language model, so that the model can effectively learn the subtle semantic differences between positive and negative samples, thereby enhancing the model's performance in processing combined reasoning tasks.

[0055] Embodiment 3: An embedded visual-language reasoning system, which is integrated in a hardware device and is suitable for real-time processing scenarios. It is mainly used in resource-constrained environments, such as edge computing devices, mobile terminals, etc., and can complete complex visual-language reasoning tasks locally without relying on cloud computing. It includes the optimization system described in Embodiment 2, and also includes:

[0056] The data acquisition module is used to acquire images from cameras or sensors and generate corresponding real-time text descriptions. Combining the text negative sample generation strategy and the visual negative feature generation module, it can perform fine-grained analysis on the real-time acquired images and texts and generate visual negative features that reflect semantic shifts.

[0057] The real-time reasoning and feedback module is responsible for reasoning based on the generated visual negative features and providing real-time feedback to the user or system based on the reasoning results. Reasoning tasks include but are not limited to complex semantic analysis tasks such as object detection, relationship recognition, and scene understanding.

[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Other modifications or equivalent substitutions made to the technical solution of the present invention by ordinary technicians in the field should be included in the scope of the claims of the present invention as long as they do not depart from the spirit and scope of the technical solution of the present invention.

Claims

1. A visual feature generation method for improving combined reasoning of visual language models, characterized in that: include: Construct a dataset where the data is an image-text pair consisting of an image and its text description; Generate negative descriptions of text by exchanging the positions or attributes of entities; By generating a negative description of the text, obtaining a semantic offset between the positive description and the negative description to generate a difference vector; Generating visual disturbances through the difference vector, guiding image feature modification, and obtaining visual features that maintain semantic consistency with the negative description of the text; The generated visual negative features are applied to contrastive learning of vision-language models to optimize the performance of the model in combinatorial reasoning tasks.

2. The method for generating visual features for improving combined reasoning of visual language models according to claim 1, characterized in that: Methods for constructing a dataset where the data are image-text pairs consisting of images and their text descriptions include: Construct a data set, where the data is an image-text pair consisting of an image and its text description. orig Parsed into a scene graph representation, for the text T orig Parse to obtain the triple (O1, R, O2) or attribute pairs (A1, O1) and (A2, O2); Among them, O1 and O2 represent the text T orig Different entities in, A1 and A2 represent the attribute information corresponding to the entity.

3. The method for generating visual features for improving combined reasoning of visual language models according to claim 2, characterized in that: Methods for generating negative descriptions of text by exchanging the positions or attributes of entities include: Assume that after parsing, a triple (O1, R, O2) is obtained, and the positions of entities O1 and O2 are swapped in the triple (O1, R, O2) by Swap((O1, R, O2)) = (O2, R, O1); Assume that after parsing, we get attribute pairs (A1, O1) and (A2, O2), and attribute pairs (A1, O1) and (A2, O2) exchange attribute information through Swap((A1, O1), (A2, O2)). The calculation method is: Among them, Swap represents a swap operation.

4. The method for generating visual features for improving combined reasoning of visual language models according to claim 1, characterized in that: The method of obtaining the semantic offset between the positive description and the negative description to generate a difference vector by generating a negative description of the text includes: Use the pre-trained vision-language model to train the text T orig and T neg Perform embedding processing to generate semantic embedding vectors of text and the semantic embedding vector of the negative description of the text Get and The semantic shift difference vector Δ e , capturing the semantic changes of text, calculation method: Among them, Δ e Represents the semantic shift from positive samples to negative samples.

5. The method for generating visual features for improving combined reasoning of visual language models according to claim 1, characterized in that: The method of generating visual disturbance through the difference vector, guiding image feature modification, and obtaining visual features that maintain semantic consistency with the negative description of the text includes: From the image I orig Extract image features Generate negative descriptions with text T neg Corresponding visual negative features The calculated difference vector Δ e With image features Combined to generate visual negative features Calculation method:

6. An optimization system for improving the performance of visual language model combination reasoning tasks, used to execute the visual feature generation method according to any one of claims 1 to 5, characterized in that: include: A data set building module is used to build a data set for generating negative samples; A visual negative feature generation module for extracting semantic embeddings of text and negative descriptions using a pre-trained vision-language model; A model fine-tuning module is used to use the generated visual negative features, text features of the image, and image features for fine-tuning the visual-language model.

7. An embedded vision-language reasoning system, characterized in that: The optimization system according to claim 6 further comprises: A data acquisition module is used to acquire images from a camera or sensor and generate corresponding real-time text descriptions; The real-time reasoning and feedback module is used to perform reasoning based on the generated visual negative features and provide real-time feedback to the user or system based on the reasoning results.

Images