Semantic alignment method and system in multi-graph multi-mode large language model

By introducing a bidirectional boot mechanism and adaptive weight adjustment module of contextual images into the multimodal large language model, the semantic aberration problem between multiple images is solved, and high-performance semantic alignment in scenarios with large image differences is achieved. The performance of the prior art.

CN119027945AActive Publication Date: 2024-11-26ZHEJIANG UNIV +1

Patent Information

Application Number
CN202410956798.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-17
Publication Date
2024-11-26
Estimated Expiration
2044-07-17

AI Technical Summary

Technical Problem

The existing multimodal large models lack semantic alignment when processing multiple images, making it difficult to establish meaningful associations and generate accurate answers when there are significant differences between images.

Method used

By introducing contextual semantic guidance for contextual images at the perception stage, the two-way guide mechanism is used to align visual markers before the comprehensive analysis step, and extract contextual semantic features highly correlated with the current perceived image through the adaptive weight adjustment module to generate enhanced vision mark.

Benefits of technology

37% and 22% performance improvements were achieved on Group Captioning and Storytelling tasks, surpassing commercial closed-source models such as GPT-4V and GeminiPro, and performing excellently in scenarios with large image differences and similar images In the scenarios, high semantic alignment capabilities are also maintained.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027945A_ABST
    Figure CN119027945A_ABST
Patent Text Reader

Abstract

The invention discloses a semantic alignment method and system in a multi-graph multi-modal large language model, and the method specifically comprises the steps: collecting a group slice group from an existing single-graph data set, and constructing a multi-graph multi-modal sample through image segmentation, an image filling technology and a large language model; and the method is applied to multi-graph and multi-mode large model training with semantic alignment enhancement. According to the model, firstly, an input picture is encoded through a visual encoder, a visual encoding graph is generated, and a visual mark is extracted through a Q-former module. On this basis, context semantic features are generated through an adaptive weight adjustment module, and the context semantic features are combined with semantic guide features of the current perceptual picture to finally generate a visual mark with enhanced semantic alignment. According to the method, the semantic alignment capability in a multi-picture task is remarkably improved, and particularly, relatively high performance improvement is achieved in a scene with relatively large picture processing difference; and meanwhile, a certain semantic alignment capability can be kept under the condition that the pictures are highly similar.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of multimodal signal processing, and in particular relates to a semantic alignment method and system in a multi-graph multimodal large language model. Background Art

[0002] Multimodal large language models (MLLMs), with GPT-4V being the main representative, have shown great potential in handling multi-image cross-modal instructions. Reviewing the usage reports of GPT-4V, it is observed that it mainly focuses on highly similar images when handling multi-image tasks. In these cases, images share almost the same background context, so multimodal large models can effectively solve reasoning tasks by aligning similarities and locating differences between visual content. However, when faced with images that differ significantly in content, context, or style, the effectiveness of multimodal large models, including GPT-4V, decreases. In these cases, large models may have difficulty establishing meaningful semantic associations between images and generating accurate responses.

[0003] After the above analysis, this defect can be attributed to the lack of semantic alignment of existing multimodal large language models when processing multiple images. Specifically, the current multimodal large language models include two stages when processing multiple images:

[0004] Independent Perception: This step uses visual perceptrons (such as Q-Former, linear layers, and Resampler) to map the visual features encoded by the pre-trained visual backbone network to visual tags in the feature space of the large language model. In this way, the visual tags of the image received by the large model will contain its visual features.

[0005] Comprehensive Analysis: After obtaining the visual tags for each image, the model fuses them with the textual tags of the instructions using a cross-modal attention mechanism. This step allows the model to combine information from different modalities and images.

[0006] In this way, multimodal large models mainly deal with multi-image reasoning in the comprehensive analysis stage, which leads to the problem of semantic misalignment and hinders its ability to discover associations between images. This is because the visual tags used in the comprehensive analysis step may lack the information needed to "link" multiple images to identify associations, which is particularly evident when the input images show large differences. Specifically, visual tags are affected by the inductive bias of training data (such as image-caption pairs), causing them to selectively express salient visual content. Therefore, in complex contexts, the focus of visual tags for different images may be different, which hinders semantic alignment. In addition, in the comprehensive analysis stage, the large model jointly calculates attention for visual tags and text tags, which causes the interaction between visual tags to be weakened in a large number of tag interactions, making it difficult to perform semantic alignment.

[0007] The solution to the above semantic misalignment problem is to introduce contextual semantics from context images as guidance during the perception stage. By utilizing the context image as a reference, the visual markers of the current perceived image can be accurately aligned with the contextual semantics before the comprehensive analysis step. However, it should be noted that all context images in the input multimodal instructions may not be directly related to the current perceived image due to the presence of irrelevant or even noisy information. In this case, directly incorporating all the information of all context images may pose challenges to the perception process. Therefore, in addition to the above semantic misalignment problem, how to accurately extract contextual semantics that are highly relevant to the current perceived image is also an important issue that deserves careful consideration. Summary of the invention

[0008] The purpose of the present invention is to solve the problems existing in the prior art and provide a semantic alignment method and system in a multi-graph multi-modal large language model.

[0009] In order to achieve the above-mentioned invention object, the present invention specifically adopts the following technical solutions:

[0010] In a first aspect, the present invention provides a semantic alignment method in a multi-graph multi-modal large language model, comprising the following steps:

[0011] S1. Construct multi-image multimodal samples from existing single-image datasets, each of which includes several final images and final description text corresponding to each multi-image multimodal sample; wherein each final image includes the same object, and the same object in each final image is presented in different poses, lighting conditions or viewing angles;

[0012] S2. Obtain a trained semantic alignment model, input the prompt words containing the task to be completed and the picture to be semantically aligned into the trained semantic alignment model, and the semantic alignment model outputs the answer corresponding to the prompt words of the task to be completed;

[0013] The semantic alignment model includes a visual encoder, a bidirectional guidance module and a large language model, the bidirectional guidance module includes two Q-former modules and an adaptive weight adjustment module, the first Q-former module and the second Q-former module interact through a linear layer, and the first Q-former module and the second Q-former module share parameters;

[0014] The semantic alignment model is trained on a multi-image multimodal dataset generated by multi-image multimodal samples. During the training process of the semantic alignment model, the parameters of the visual encoder, the first Q-former module, the second Q-former module and the large language model are fixed and not updated, and only the adaptive weight adjustment module and the linear layer parameters for interaction are updated.

[0015] Based on the above scheme, each step can be implemented in the following preferred specific manner.

[0016] As a preferred embodiment of the first aspect, the specific process of step S1 is as follows:

[0017] S11. Extracting pictures of different characters from the single-image dataset of characters and constructing several picture groups, each picture group consists of two character pictures, wherein a character picture in one picture group contains the same character as a character picture in another picture group, a character picture in another picture group contains a different character from a character picture in another picture group, and pictures containing the same character are pictures of the same character at different angles;

[0018] S12. Use Segment Anything to segment the images in each image group, and obtain a segmented image group corresponding to each image group;

[0019] S13. Preprocessing each group of segmented picture groups, each group of segmented picture groups corresponding to a group of preprocessed picture groups, placing two pictures in each group of preprocessed picture groups on the same background picture, each group of preprocessed picture groups corresponding to a mask picture; wherein the pixel value of each pixel point on the background picture is 0;

[0020] S14. Randomly select one of the diverse picture background descriptions generated by the large language model as the picture background description of each mask picture, input the mask pictures and their corresponding picture background descriptions into the Stable-Diffusion-Inpainting model to complete the filling process, and each mask picture corresponds to a final picture that conforms to the picture background description and has a harmonious overall picture;

[0021] S15. Use the large language model to describe the same characters in each group of pictures obtained in S11 to obtain an initial description; input the prompt words used to guide the large language model to optimize the initial description into the large language model, and the large language model optimizes the initial description to obtain a final description text.

[0022] As a preferred embodiment of the above-mentioned first aspect, in step S13, when preprocessing each group of segmented picture groups, each group of segmented picture groups contains two segmented character pictures, the height of each segmented character picture is scaled to within a preset range and the width is scaled proportionally, and each segmented character picture corresponds to a scaled character picture, and a preprocessed picture group is generated by the scaled character pictures.

[0023] As a preferred embodiment of the first aspect, in step S15, the large language model used to describe the character adopts InstructBLIP, and the large language model used to optimize the initial description adopts ChatGPT.

[0024] As a preferred embodiment of the first aspect, in the semantic alignment model, the specific processing flow is as follows:

[0025] S21. Inputting the image to be semantically aligned into the visual encoder for encoding to obtain a visual encoding map;

[0026] S22. After encoding the prompt word of the task to be completed, a text tag is obtained, and the pre-trained query vector and the text tag are concatenated to obtain an input query vector;

[0027] S23. When the semantic alignment model processes the i-th visual coding map, the i-th visual coding map and the input query vector are input into the first Q-former module, and the query vector output by the (l-1)th layer of the first Q-former module passes through the first linear layer to obtain the semantic guidance feature of the current perceived image;

[0028] S24. The remaining visual coding maps except the i-th visual coding map are input into the adaptive weight adjustment module, and the adaptive weight adjustment module first generates normalized adaptive weights according to the remaining visual coding maps, and then the remaining visual coding maps are weighted with their corresponding adaptive weights to obtain weighted visual coding maps, and finally the weighted visual coding maps are summed to obtain a fused visual coding map;

[0029] S25. input the fused visual encoding map and the input query vector into the second Q-former module, add the query vector output by the (l-1)th layer of the second Q-former module to the semantic guidance feature of the current perceived image, and obtain an updated output query vector;

[0030] S26. The remaining layers of the second Q-former module extract relevant details from the fused visual encoding map, and obtain contextual semantic features by passing the query vector output by the k≥lth layer of the second Q-former module through the second linear layer;

[0031] S27. Add the query vector and contextual semantic features output by the (k-1)th layer of the first Q-former module as the kth layer input of the first Q-former module; generate a final visual markup of the current perceived image with enhanced semantic alignment from the kth layer and the remaining layer outputs of the first Q-former module;

[0032] S28. The semantic alignment model processes each visual coding map until all the visual coding maps are processed, and the final visual tags and text tags corresponding to each visual coding map are input into the large language model, and the large language model outputs the final answer.

[0033] As a preferred embodiment of the first aspect, the adaptive weight adjustment module generates a normalized adaptive weight The specific implementation is as follows:

[0034]

[0035] Among them, Linear represents the linear layer, I m,j represents the remaining visual coding map except the i-th visual coding map; softmax represents the softmax function.

[0036] In a second aspect, the present invention provides a semantic alignment system in a multi-graph multi-modal large language model, comprising:

[0037] A data acquisition module is used to construct a multi-image multimodal sample from an existing single-image dataset, each of which includes a number of final images and a final description text corresponding to each multi-image multimodal sample; wherein each final image includes the same object, and the same object in each final image is presented in different postures, lighting conditions or viewing angles;

[0038] A result acquisition module is used to obtain a trained semantic alignment model, input the prompt words containing the task to be completed and the picture to be semantically aligned into the trained semantic alignment model, and the semantic alignment model outputs the answer corresponding to the prompt words of the task to be completed;

[0039] The semantic alignment model includes a visual encoder, a bidirectional guidance module and a large language model, the bidirectional guidance module includes two Q-former modules and an adaptive weight adjustment module, the first Q-former module and the second Q-former module interact through a linear layer, and the first Q-former module and the second Q-former module share parameters;

[0040] The semantic alignment model is trained on a multi-image multimodal dataset generated by multi-image multimodal samples. During the training process of the semantic alignment model, the parameters of the visual encoder, the first Q-former module, the second Q-former module and the large language model are fixed and not updated, and only the adaptive weight adjustment module and the linear layer parameters for interaction are updated.

[0041] In a third aspect, the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, can implement the semantic alignment method in a multi-graph multi-modal large language model as described in any one of the solutions of the first aspect above.

[0042] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for semantic alignment in a multi-graph multi-modal large language model as described in any one of the schemes of the first aspect above can be implemented.

[0043] In a fifth aspect, the present invention provides a computer electronic device, comprising a memory and a processor;

[0044] The memory is used to store computer programs;

[0045] The processor is used to implement the semantic alignment method in the multi-graph multi-modal large language model as described in any one of the solutions of the first aspect above when executing the computer program.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] (1) This paper introduces bidirectional guidance from contextual images to perform semantic alignment between multiple images. Experimental results show that the performance of group captioning and storytelling tasks is improved by 37% and 22% respectively. Even with only 7 billion parameters, the performance exceeds that of commercial closed-source large models, such as GPT-4V and GeminiPro.

[0048] (2) The proposed method can not only perform semantic alignment when the images are quite different, but also has relatively stable performance when the images are highly similar. In the Change captioning task, the performance still exceeds the existing open source multimodal large model, achieving a 39% performance improvement. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of the steps of the present invention;

[0050] Figure 2A schematic diagram of constructing a multi-image multi-modal sample for the present invention;

[0051] Figure 3 It is a schematic diagram of the structure of the semantic alignment model of the present invention;

[0052] Figure 4 It is a system block diagram of the present invention. DETAILED DESCRIPTION

[0053] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0054] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0055] The technical problem to be solved by the present invention is that the existing multimodal large model usually adopts a two-stage strategy of "independent perception-comprehensive analysis" when processing image-text instructions. However, in multi-image scenarios, especially when there are significant differences between images, the visual tags extracted in the independent perception stage will cause semantic misalignment between different images due to the lack of contextual images as guidance. Such semantically misaligned visual tags will cause great misleading in the subsequent comprehensive analysis stage, making it impossible for the large model to correctly understand the relationship between images, thereby generating inaccurate answers.

[0056] In a preferred embodiment of the present invention, Figure 1 As shown, in a preferred implementation of the present invention, the semantic alignment method in the multi-graph multi-modal large language model includes the following steps S1 to S2. The specific implementation process is described in detail below.

[0057] The present invention mainly includes two parts: one is the generation of multi-image multimodal samples based on image filling technology; the other is semantic alignment based on bidirectional guidance of contextual images.

[0058] S1. Construct multi-image multimodal samples from existing single-image datasets, each of which contains several final images and a final description text corresponding to each multi-image multimodal sample; wherein each final image contains the same object, and the same object in each final image is presented in different postures, lighting conditions, or viewing angles.

[0059] It should be noted that step S1 of the present invention is a multi-image multimodal sample generation process based on image filling technology. Most existing multi-image datasets for fine-tuning multimodal large language models may show significant similarities between images of multimodal instructions, lacking associated images with significant differences (e.g., different contexts, different styles). To this end, the present invention constructs a new large-scale multimodal dataset called MmLINK based on image filling technology for multimodal large model research. The dataset contains 69,000 multi-image multimodal samples, which are specifically used to enhance the model's cross-modal multi-image semantic alignment and association mining capabilities. Each multi-image multimodal sample contains two final images I1, I2 and a final description text T of the alignment common points. The two final images corresponding to each final image will contain the same object, but presented in different postures, lighting conditions or perspectives. The object in the present invention refers to the associated semantics contained in the image.

[0060] It is worth noting that in order to ensure the diversity of multi-image and multi-modal sample objects in the MmLINK dataset, the present invention collects images containing a variety of objects from different image sources, covering people, furniture items, icons, and book covers. For different types of objects, the present invention uses slightly different processes to create corresponding samples.

[0061] In this embodiment, if Figure 2 As shown in the figure, taking "character" as an example, the process of constructing multi-image multi-modal samples is explained. The generation process of multi-image multi-modal samples includes two stages: mask image generation and background filling. The details are as follows:

[0062] 1) Mask image generation

[0063] S11. Extract pictures of different characters from a single-image dataset of characters and construct several groups of pictures, each group of pictures consists of two characters, wherein a character picture in one group of pictures contains the same character as a character picture in another group of pictures, a character picture in another group of pictures contains a different character from a character picture in another group of pictures, and pictures containing the same character are pictures of the same character at different angles.

[0064] In this embodiment, if Figure 2As shown in the figure, pictures of different characters are extracted from the DeepFashion dataset to construct two groups of pictures. The first group of pictures [C1, C2] and the second group of pictures [C1′, C3] contain the same characters. C1 and C1′ are two pictures of the same character from different angles, and C2 and C3 are pictures of two other different characters. In this way, the two groups of pictures contain the same character, but presented in different postures, lighting conditions or perspectives; and each group of pictures has an additional character as an interference item to increase the difficulty of the task. In the subsequent steps, the pictures of these characters are used to form two mask pictures.

[0065] S12. Use Segment Anything to segment the images in each group of images, and obtain a segmented image group corresponding to each group of images.

[0066] In this embodiment, if Figure 2 As shown, since the character pictures in the DeepFashion dataset have background noise, the four character pictures in the two groups of picture images need to be segmented to remove the background noise. Specifically, Segment Anything is used to segment the four character pictures (C1, C1′, C2, C3) obtained in step S11. Because the backgrounds of the pictures in the DeepFashion dataset are all light-colored and the characters are more prominent, the one with the highest confidence is selected from the many segmentation results output by Segment Anything as the segmented character picture. In this embodiment, corresponding to the four character pictures obtained in the above steps, a total of four segmented character pictures are obtained after segmentation, and the four segmented character images are restored to the segmented picture group according to the combination of the original picture groups. Corresponding to the two groups of picture groups in the above process, after segmentation, the first group of segmented picture groups is The second group is the group of images after segmentation. is the character picture obtained after segmentation of C1, is the character picture obtained after segmentation of C2. is the character picture obtained after segmentation of C1′, This is the character picture obtained after segmentation of C3.

[0067] S13. Preprocess each group of segmented picture groups, and obtain a corresponding preprocessed picture group for each group of segmented picture groups. Place the two pictures in each preprocessed picture group on the same background picture, and obtain a corresponding mask picture for each preprocessed picture group; wherein the pixel value of each pixel point on the background picture is 0.

[0068] It should be noted that in step S13 of the present invention, when preprocessing each group of segmented picture groups, each group of segmented picture groups contains two segmented character pictures, the height of each segmented character picture is scaled to within a preset range and the width is scaled proportionally, and each segmented character picture corresponds to a scaled character picture, and a preprocessed picture group is generated by the scaled character pictures.

[0069] In this embodiment, if Figure 2 As shown, considering that the character images are usually "thin and tall", that is, the height is much greater than the width, the present invention scales the height of each segmented character image to a random number between 200 and 250, and scales the width in proportion, thereby obtaining a scaled character image. This ensures that the character image can be completely placed on the black background image. Then place the two scaled character images of the same group on a 512×512 background image, thus forming a mask image. Corresponding to the two groups of images in the above process, after the mask generation process, a total of two mask images are obtained.

[0070] 2) Background fill

[0071] In the background filling process, the mask image obtained in the previous step is filled with various background pixels to make it a complete final image, and the corresponding final description text is generated for each image group.

[0072] S14. Randomly select one from the diverse picture background descriptions generated by the large language model as the picture background description of each mask picture, input the mask pictures and their corresponding picture background descriptions into the Stable-Diffusion-Inpainting model to complete the filling process, and each mask picture corresponds to a final picture that conforms to the picture background description and has a harmonious overall picture.

[0073] In this embodiment, if Figure 2 As shown, ChatGPT is first requested to obtain 500 diverse image background descriptions for filling the mask image. For each mask image, a background description is randomly selected, and the Stable-Diffusion-Inpainting model is used to complete the filling process. The model receives the mask image and the image background description as input, and fills the mask image with more harmonious background pixels that conform to the image background description. In this way, the mask image is filled into the final image. Corresponding to the two mask images in the above process, after the mask generation process, two final images I1 and I2 are obtained.

[0074] S15. Use the large language model to describe the same characters in each group of pictures obtained in S11 to obtain an initial description; input the prompt words used to guide the large language model to optimize the initial description into the large language model, and the large language model optimizes the initial description to obtain the final description text T.

[0075] In S15 of the present invention, the large language model used to describe the character role and the large language model used to optimize the initial description can be the same or different. Figure 2 As shown, the InstructBLIP model is first used to describe the same character in each group of pictures in the above steps to obtain an initial description, and then ChatGPT is used to optimize the initial description to obtain the final description text T. Among them, the initial description aims to describe the similarities between two pictures of the same character from different angles (such as C1, C1′), that is, to describe the character.

[0076] S2. Obtain a trained semantic alignment model, input the prompt words containing the task to be completed and the picture to be semantically aligned into the trained semantic alignment model, and the semantic alignment model outputs the answer corresponding to the prompt words of the task to be completed;

[0077] The semantic alignment model includes a visual encoder, a bidirectional guidance module and a large language model, the bidirectional guidance module includes two Q-former modules and an adaptive weight adjustment module, the first Q-former module and the second Q-former module interact through a linear layer, and the first Q-former module and the second Q-former module share parameters;

[0078] The semantic alignment model is trained on a multi-image multimodal dataset generated by multi-image multimodal samples. During the training process of the semantic alignment model, the parameters of the visual encoder, the first Q-former module, the second Q-former module and the large language model are fixed and not updated, and only the adaptive weight adjustment module and the linear layer parameters for interaction are updated.

[0079] It should be noted that step S2 of the present invention is a semantic alignment process based on bidirectional guidance of context pictures. Visual markers are extracted for the current perception picture and the context picture respectively, and mutual guidance is performed in the process of extracting the visual markers. Specifically, in order to cope with the semantic alignment problem existing in the existing multimodal large model when processing images with different contexts, the present invention introduces a bidirectional semantic guidance mechanism in the image perception stage, generates visual markers with enhanced semantic alignment from patch-level features, and constructs a semantic alignment model based on this. In the comprehensive analysis stage, the large language model of the semantic alignment model processes the visual markers and the input text markers and generates the final prediction results. The bidirectional semantic guidance mechanism mainly includes two interactive processes: auxiliary visual marker extraction and context semantic generation. Through the bidirectional semantic guidance mechanism based on context pictures, the semantic alignment model can promote the semantic alignment of multiple pictures. The overall process of the semantic alignment model can be divided into four steps.

[0080] A) Image feature generation

[0081] S21. Input the image to be semantically aligned into the visual encoder for encoding to obtain a visual encoding map.

[0082] It should be noted that, in step S21 of the present invention, Figure 3 As shown in the figure, given N input images, the visual encoder is used to convert these input images into patch-level features to obtain a visual encoding map. The i-th input image is used as the current perception image, I i = {I i,j} P represents P corresponding visual coding graphs, where P is the number of patches and j represents the patch index. In this embodiment, the visual encoder adopts the EVA-CLIP model.

[0083] S22. After encoding the prompt word of the task to be completed, a text tag is obtained, and the pre-trained query vector and the text tag are concatenated to obtain an input query vector.

[0084] It should be noted that, in step S22 of the present invention, Figure 3 As shown, the text is marked Figure 3 The green circle in the figure shows the pre-trained query vector. Figure 3 The blue marker circle in .

[0085] B) Assisted Visual Marker Extraction

[0086] First, in the auxiliary visual tag extraction process, the semantic alignment model applies the first Q-former module to process the current perceived image based on the natural language query and extract the initial visual tag, that is, the semantic guidance feature h of the current perceived image. iNext, in the context semantic generation process, the present invention uses another Q-former module as a new visual tag generator. i Under the guidance of , the second Q-former module extracts the synchronous context semantics from the context image and obtains the context semantic feature c i Finally, the contextual semantics are passed back to the first Q-former module to guide the update of visual tags based on natural language queries, enhancing the semantic alignment of visual tags by aligning the contextual information.

[0087] The specific steps of the auxiliary visual marker extraction process are as follows:

[0088] S23. When the semantic alignment model processes the i-th visual encoding map, the i-th visual encoding map and the input query vector are input into the first Q-former module. The query vector output by the (l-1)th layer of the first Q-former module passes through the first linear layer to obtain the semantic guidance feature h of the current perceived image. i .

[0089] It should be noted that step S23 of the present invention is intended to provide guidance from the current perceived image to the context image, so that the subsequent steps can extract information highly related to the current perceived image from the context image for guidance. i The present invention uses the Q-former module proposed in InstructBLIP as the first Q-former module to perform semantic compression and extract visual markers (the semantic guidance features h of the current perceived image). i ). The first Q-former module is composed of several cross-attention layers stacked together, which initially takes the pre-trained query vector and text tag t as the query, and the visual encoding diagram I i As the key and value of cross-attention, the output query vector is updated through each cross-attention layer. It can be expressed as:

[0090]

[0091] Among them, Q l-1 is the (l-1)th layer of the first Q-former module; represents the output query vector of the (l-1)th layer of the first Q-former module (that is, the query vector input to the lth layer of the first Q-former module); represents the output query vector of the (l-2)th layer of the first Q-former module (that is, the query vector input to the (l-1)th layer of the first Q-former module).

[0092] In order to extract the visual mark of the current perceived image for the subsequent guidance process, the output query vector of the (l-1) layer of the first Q-former module is obtained. After that, it passes through the linear layer and is used as the semantic guidance feature of the current perceived image:

[0093]

[0094] C) Contextual semantic generation

[0095] S24. The remaining visual coding maps except the i-th visual coding map are input into the adaptive weight adjustment module. The adaptive weight adjustment module first generates normalized adaptive weights according to the remaining visual coding maps, and then weights the remaining visual coding maps with their corresponding adaptive weights to obtain weighted visual coding maps. Finally, the weighted visual coding maps are summed to obtain a fused visual coding map.

[0096] It should be noted that, in step S24 of the present invention, the specific implementation method of the adaptive weight adjustment module generating the normalized adaptive weight is as follows:

[0097]

[0098] Among them, Linear represents the linear layer, I m,j represents the remaining visual coding map except the i-th visual coding map; softmax represents the softmax function.

[0099] In this embodiment, if Figure 3 As shown in Figure 1, after obtaining the semantic guidance features of the current perception picture, it is necessary to use it to guide the context picture, thereby filtering out the redundant information in the context picture and extracting the information in the context picture that is highly relevant to the current perception picture. Specifically, the pictures other than the current perception picture are taken as context pictures, and the set of context pictures is represented as {I m} m≠i , the patch-level feature (visual encoding map) corresponding to the mth (m≠i) context image is expressed as Use the adaptive weight adjustment module to assign weights to each context image patch to reduce the influence of irrelevant patches. Generate normalized patch-level weights through linear layers and softmax functions Then re-weight the visual encoding map of each context image and sum the weighted visual encoding maps to obtain the fused visual encoding map This process reduces the impact of irrelevant patches and enhances the fused visual encoding map. Patch-level alignment with the current perceived image.

[0100] S25. Input the fused visual encoding map and the input query vector into the second Q-former module, and convert the query vector w output by the (l-1)th layer of the second Q-former module into l and the semantically guided feature h of the current perceived image i Add together to get the updated output query vector

[0101] It should be noted that in step S25 of the present invention, the auxiliary visual marker extraction process requires the contextual semantic features of the context image to guide the visual marker extraction of the current perceived image. However, it should be noted that all context images in the input may not be directly related to the current perceived image and the natural language query. In addition, even highly relevant images may contain irrelevant or noisy information. In order to solve these problems, the present invention reuses the structure of the Q-former module in the context semantic generation process, constructs a second Q-former module, and extracts semantics from the fused visual coding map. The second Q-former module has the same structure and parameters as the first Q-former module. The second Q-former module also uses the pre-trained query vector and text tag t as the Query. Unlike the first Q-former module, the second Q-former module uses the fused visual coding map As the Key and Value of cross-attention, the process of extracting semantics is as follows:

[0102]

[0103] Among them, W l-1 is the (l-1)th layer of the second Q-former module; w l represents the output query vector of the (l-1)th layer of the second Q-former module (that is, the query vector input to the lth layer of the second Q-former module); w l-1 represents the output query vector of the (l-2)th layer of the second Q-former module (that is, the query vector input to the (l-1)th layer of the second Q-former module).

[0104] In the lth layer of the second Q-former module, the semantic guidance feature h of the current perceived image is obtained i Add to get the updated output query vector

[0105]

[0106] S26. The remaining layers of the second Q-former module are derived from the fused visual encoding map Extract relevant details from the query vector w output by the k≥lth layer of the second Q-former module k+1 After the second linear layer, we get the contextual semantic feature c i .

[0107] It should be noted that in step S26 of the present invention, in the subsequent k≥l layers of the second Q-former module, the semantic guidance feature h of the current perceived image is i Under the guidance of By effectively reducing the redundant details of the context image in the multimodal instructions, the second Q-former module can generate contextual semantic features that are highly relevant to the current perceived image. Specifically, the query vector w output from the kth layer of the second Q-former module through the second linear layer is k+1 Extract the context semantics from the i :

[0108] c i =Linear(w k+1 )

[0109] The generated contextual semantic features aggregate information from the entire context that is relevant to the current perceived image, facilitating the generation of semantically aligned visual tags.

[0110] D) Final visual markup generation

[0111] S27. The query vector output by the (k-1)th layer of the first Q-former module and contextual semantic features c i Add as the k-th layer input of the first Q-former module The final visual markup of the current perceived image with enhanced semantic alignment is generated by the kth layer and the remaining layers output of the first Q-former module.

[0112] It should be noted that in step S27 of the present invention, the contextual semantic features obtained are used to guide the final visual mark generation process of the current perceived image, thereby promoting the semantic alignment of the current perceived image and the contextual image. Specifically, after obtaining the contextual semantic features, the kth layer subsequent to the first Q-former module is selected, and the contextual semantic features c i Added to the input of the kth layer of the first Q-former module for guidance:

[0113]

[0114] Subsequent layers of the first Q-former module will refer to the contextual semantics to generate the final visual tags of the current perceived image with enhanced semantic alignment. These final visual tags with enhanced semantic alignment will be fed into the large language model to generate the final answer together with the text tags.

[0115] S28. The semantic alignment model processes each visual coding map until all the visual coding maps are processed, and the final visual tags and text tags corresponding to each visual coding map are input into the large language model, and the large language model outputs the final answer.

[0116] It should be noted that, in the present invention, the layer indexes k and l can select a layer in the first Q-former module and the second Q-former module for interaction according to actual conditions, as long as k≥l is satisfied. In the embodiment of the present invention, k and l are set to the same value, both representing the last layer of the first Q-former module and the second Q-former module, that is, two interactions are performed at the final layer of the first Q-former module and the second Q-former module, and l=k=12.

[0117] It should also be noted that in the present invention, for efficient training, the parameters of the visual encoder and the large language model are frozen. In addition, the first Q-former module and the second Q-former module are also frozen. This means that only two linear layers for the first Q-former module and the second Q-former module to interact and the adaptive weight adjustment module need to be fine-tuned, with a total of only 4.3 million (0.05%) trainable parameters. The present invention uses the AdamW optimizer to train the semantic alignment model, and sets the parameters to β = (0.9, 0.999); a learning rate of 2e-5 and a batch size of 20 are used for training.

[0118] In order to better demonstrate the specific implementation and technical effects of the present invention, the semantic alignment method in the multi-graph multi-modal large language model shown in steps S1 to S2 in the above preferred implementation is applied to a specific example.

[0119] Example

[0120] The specific implementation process of the semantic alignment method in the multi-graph multi-modal large language model used in this embodiment is as described above and will not be repeated here.

[0121] In order to demonstrate the technical effect of this embodiment, the present invention is verified on the Group Captioning and Storytelling tasks. The Group Captioning task requires the model to discover the common features of a group of pictures and describe them in natural language. The present invention is verified on three public datasets: Conceptual, Animal, and Vehicle. The Storytelling task requires the model to continue the story based on the content of the last picture given a number of pictures and the corresponding storyline. The present invention is verified on three public datasets: AESOP, VIST, and DM800K.

[0122] In order to objectively evaluate the performance of the method in this embodiment, the performance indicators of the above two tasks and six data sets are all the commonly used indicators ROUGE-L, CIDEr and BLEU-4. The experimental results are shown in Tables 1 and 2. In Tables 1 and 2, R represents ROUGE-L, C represents CIDEr, and B represents BLEU-4.

[0123] Table 1 Results of the present invention on the Group Captioning task

[0124]

[0125] Table 2 Results of the present invention on the Storytelling task

[0126]

[0127] From the above results, it can be seen that in all tasks, all data sets, and all indicators, this example surpasses open source multimodal large models, closed source commercial multimodal large models, including the current strongest multimodal large models such as GPT-4V and Gemini Pro. Specifically, in the CIDEr indicator, this example achieved a 37% performance improvement in the GroupCaptioning task and a 22% performance improvement in the Storytelling task compared to the current strongest model. These demonstrate the excellent multi-image semantic alignment capability of the method proposed in the present invention in scenarios with large image differences.

[0128] In order to verify that the proposed method still retains a certain semantic alignment capability in scenes where the pictures are relatively similar, this example conducted experiments on the Change Captioning task. This task requires the model to discover the differences between two highly similar pictures and describe them in natural language. The present invention was tested on four public datasets: IEdit, Spot-the-Diff, Birds-to-Words, and CLEVR-Change. The experimental results are shown in Table 3. In Table 3, R represents ROUGE-L, C represents CIDEr, and B represents BLEU-4.

[0129] Table 3 Results of the present invention on the Change Captioning task

[0130]

[0131] As can be seen from Table 3, the average indicators of this example on the Change Captioning task all exceed those of the open source multimodal large model, and the performance improvement on the CIDEr indicator is 39%. This shows that the proposed method can still retain a certain semantic alignment capability in scenarios where the backgrounds of the images are relatively similar.

[0132] It should also be noted that the semantic alignment method in the multi-graph multi-modal large language model in the above embodiment can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a semantic alignment system in a multi-graph multi-modal large language model corresponding to the semantic alignment method in the multi-graph multi-modal large language model provided in the above embodiment, such as Figure 4 As shown, it includes:

[0133] A data acquisition module is used to construct a multi-image multimodal sample from an existing single-image dataset, each of which includes a number of final images and a final description text corresponding to each multi-image multimodal sample; wherein each final image includes the same object, and the same object in each final image is presented in different postures, lighting conditions or viewing angles;

[0134] A result acquisition module is used to obtain a trained semantic alignment model, input the prompt words containing the task to be completed and the picture to be semantically aligned into the trained semantic alignment model, and the semantic alignment model outputs the answer corresponding to the prompt words of the task to be completed;

[0135] The semantic alignment model includes a visual encoder, a bidirectional guidance module and a large language model, the bidirectional guidance module includes two Q-former modules and an adaptive weight adjustment module, the first Q-former module and the second Q-former module interact through a linear layer, and the first Q-former module and the second Q-former module share parameters;

[0136] The semantic alignment model is trained on a multi-image multimodal dataset generated by multi-image multimodal samples. During the training process of the semantic alignment model, the parameters of the visual encoder, the first Q-former module, the second Q-former module and the large language model are fixed and not updated, and only the adaptive weight adjustment module and the linear layer parameters for interaction are updated.

[0137] It is understandable that the semantic alignment method in the multi-graph multi-modal large language model described in S1 to S2 above can be substantially implemented by a computer program. Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer program product corresponding to the semantic alignment method in the multi-graph multi-modal large language model provided in the above embodiment, which includes a computer program / instruction. When the computer program / instruction is executed by a processor, the semantic alignment method in the multi-graph multi-modal large language model described in the above embodiment can be implemented.

[0138] Similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the semantic alignment method in the multi-graph multi-modal large language model provided in the above embodiment, which includes a memory and a processor;

[0139] The memory is used to store computer programs;

[0140] The processor is used to implement the semantic alignment method in the multi-graph multi-modal large language model in the above embodiment when executing the computer program.

[0141] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0142] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the semantic alignment method in the multi-graph multi-modal large language model provided in the above embodiment, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the semantic alignment method in the multi-graph multi-modal large language model in the above embodiment can be implemented.

[0143] It is understandable that the above storage medium may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. The storage medium may also be a U disk, a mobile hard disk, a magnetic disk or an optical disk, etc., which can store program codes.

[0144] It is understandable that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0145] It should also be noted that those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0146] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A semantic alignment method in a multi-graph multi-modal large language model, characterized in that: The following steps are involved: S1. Construct multi-image multimodal samples from existing single-image datasets, each of which includes several final images and final description text corresponding to each multi-image multimodal sample; wherein each final image includes the same object, and the same object in each final image is presented in different poses, lighting conditions or viewing angles; S2. Obtain a trained semantic alignment model, input the prompt words containing the task to be completed and the picture to be semantically aligned into the trained semantic alignment model, and the semantic alignment model outputs the answer corresponding to the prompt words of the task to be completed; The semantic alignment model includes a visual encoder, a bidirectional guidance module and a large language model, the bidirectional guidance module includes two Q-former modules and an adaptive weight adjustment module, the first Q-former module and the second Q-former module interact through a linear layer, and the first Q-former module and the second Q-former module share parameters; The semantic alignment model is trained on a multi-image multimodal dataset generated by multi-image multimodal samples. During the training process of the semantic alignment model, the parameters of the visual encoder, the first Q-former module, the second Q-former module and the large language model are fixed and not updated, and only the adaptive weight adjustment module and the linear layer parameters for interaction are updated.

2. The semantic alignment method in a multi-graph multi-modal large language model according to claim 1, characterized in that: The specific process of step S1 is as follows: S11. Extracting pictures of different characters from the single-image dataset of characters and constructing several picture groups, each picture group consists of two character pictures, wherein a character picture in one picture group contains the same character as a character picture in another picture group, a character picture in another picture group contains a different character from a character picture in another picture group, and pictures containing the same character are pictures of the same character at different angles; S12. Use Segment Anything to segment the images in each image group, and obtain a segmented image group corresponding to each image group; S13. Preprocessing each group of segmented picture groups, each group of segmented picture groups corresponding to a group of preprocessed picture groups, placing two pictures in each group of preprocessed picture groups on the same background picture, each group of preprocessed picture groups corresponding to a mask picture; wherein the pixel value of each pixel point on the background picture is 0; S14. Randomly select one of the diverse picture background descriptions generated by the large language model as the picture background description of each mask picture, input the mask pictures and their corresponding picture background descriptions into the Stable-Diffusion-Inpainting model to complete the filling process, and each mask picture corresponds to a final picture that conforms to the picture background description and has a harmonious overall picture; S15. Use the large language model to describe the same characters in each group of pictures obtained in S11 to obtain an initial description; input the prompt words used to guide the large language model to optimize the initial description into the large language model, and the large language model optimizes the initial description to obtain a final description text.

3. The semantic alignment method in a multi-graph multi-modal large language model according to claim 2, characterized in that: In step S13, when preprocessing each group of segmented picture groups, each group of segmented picture groups contains two segmented character pictures, the height of each segmented character picture is scaled to a preset range and the width is scaled proportionally, and each segmented character picture corresponds to a scaled character picture, and a preprocessed picture group is generated from the scaled character pictures.

4. The semantic alignment method in a multi-graph multi-modal large language model according to claim 2, characterized in that: In step S15, the large language model used to describe the character adopts InstructBLIP, and the large language model used to optimize the initial description adopts ChatGPT.

5. The semantic alignment method in a multi-graph multi-modal large language model according to claim 1, characterized in that: In the semantic alignment model, the specific processing flow is as follows: S21. Inputting the image to be semantically aligned into the visual encoder for encoding to obtain a visual encoding map; S22. After encoding the prompt word of the task to be completed, a text tag is obtained, and the pre-trained query vector and the text tag are concatenated to obtain an input query vector; S23. When the semantic alignment model processes the i-th visual coding map, the i-th visual coding map and the input query vector are input into the first Q-former module, and the query vector output by the (l-1)th layer of the first Q-former module passes through the first linear layer to obtain the semantic guidance feature of the current perceived image; S24. The remaining visual coding maps except the i-th visual coding map are input into the adaptive weight adjustment module, and the adaptive weight adjustment module first generates normalized adaptive weights according to the remaining visual coding maps, and then the remaining visual coding maps are weighted with their corresponding adaptive weights to obtain weighted visual coding maps, and finally the weighted visual coding maps are summed to obtain a fused visual coding map; S25. input the fused visual encoding map and the input query vector into the second Q-former module, add the query vector output by the (l-1)th layer of the second Q-former module to the semantic guidance feature of the current perceived image, and obtain an updated output query vector; S26. The remaining layers of the second Q-former module extract relevant details from the fused visual encoding map, and obtain contextual semantic features by passing the query vector output by the k≥lth layer of the second Q-former module through the second linear layer; S27. Add the query vector and contextual semantic features output by the (k-1)th layer of the first Q-former module as the kth layer input of the first Q-former module; generate a final visual markup of the current perceived image with enhanced semantic alignment from the kth layer and the remaining layer outputs of the first Q-former module; S28. The semantic alignment model processes each visual coding map until all the visual coding maps are processed, and the final visual tags and text tags corresponding to each visual coding map are input into the large language model, and the large language model outputs the final answer.

6. The semantic alignment method in a multi-graph multi-modal large language model according to claim 5, characterized in that: The adaptive weight adjustment module generates normalized adaptive weights The specific implementation is as follows: Among them, Linear represents the linear layer, I m,j represents the remaining visual coding map except the i-th visual coding map; softmax represents the softmax function.

7. A semantic alignment system in a multi-graph multi-modal large language model, characterized in that: include: A data acquisition module is used to construct a multi-image multimodal sample from an existing single-image dataset, each of which includes a number of final images and a final description text corresponding to each multi-image multimodal sample; wherein each final image includes the same object, and the same object in each final image is presented in different postures, lighting conditions or viewing angles; A result acquisition module is used to obtain a trained semantic alignment model, input the prompt words containing the task to be completed and the picture to be semantically aligned into the trained semantic alignment model, and the semantic alignment model outputs the answer corresponding to the prompt words of the task to be completed; The semantic alignment model includes a visual encoder, a bidirectional guidance module and a large language model, the bidirectional guidance module includes two Q-former modules and an adaptive weight adjustment module, the first Q-former module and the second Q-former module interact through a linear layer, and the first Q-former module and the second Q-former module share parameters; The semantic alignment model is trained on a multi-image multimodal dataset generated by multi-image multimodal samples. During the training process of the semantic alignment model, the parameters of the visual encoder, the first Q-former module, the second Q-former module and the large language model are fixed and not updated, and only the adaptive weight adjustment module and the linear layer parameters for interaction are updated.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, it can implement the semantic alignment method in a multi-graph multi-modal large language model as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the semantic alignment method in the multi-graph multi-modal large language model according to any one of claims 1 to 6 is implemented.

10. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the semantic alignment method in the multi-graph multi-modal large language model as described in any one of claims 1 to 6 when executing the computer program.

Citation Information

Patent Citations

  • Image description model training method based on context reasoning and image description method

    CN115578726A

  • Multi-modal model training method and device, equipment and storage medium

    CN117611938A

  • Zero sample image processing system and processing method for cross-modal semantic alignment

    CN117911814A

Cited By

  • Unsupervised skeleton behavior recognition method and system based on multi-modal key semantics

    CN119785422A

  • Diffusion generation method based on implicit multi-mode guide alignment

    CN119919465A

  • Image description training method and device based on feature correction and storage medium

    CN120747708A

  • Cross-view pedestrian re-identification method based on visual language prompt learning

    CN120976969A

  • A cross-view pedestrian re-identification method based on visual language prompt learning

    CN120976969B