Semantic alignment method and system in multi-diagram multi-modal large language model

Abstract

The application discloses a kind of semantic alignment method and system in multi-graph multimodal large language model, specific process is: from existing single graph data set to collect group piece group, and utilize image segmentation, image filling technique and large language model to construct multi-graph multimodal sample, and it is used in semantic alignment enhanced multi-graph multimodal large model training.Wherein, model is encoded to input picture by visual encoder first, generates visual encoding graph, and extracts visual mark by Q-former module.On this basis, context semantic features are generated by adaptive weight adjustment module, and are combined with the semantic guiding features of the current perception picture, and finally generate visual marks with enhanced semantic alignment.The application significantly improves the semantic alignment capability in multi-graph tasks, especially in the context of handling large differences in pictures, achieving high performance improvement;At the same time, it can also ensure a certain semantic alignment capability in the case of highly similar pictures.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of multi-modal signal processing, and particularly relates to a semantic alignment method and system in a multi-image multi-modal large language model. BACKGROUND

[0002] Multi-modal large language models (MLLMs) have shown great potential in handling multi-image cross-modal instructions, with GPT-4V as the main representative. Reviewing the usage reports of GPT-4V, it is observed that it mainly focuses on highly similar images when handling multi-image tasks. In these cases, the images share almost the same background context, so the multi-modal large model can effectively solve the reasoning task by aligning the similarities and locating the differences between the visual contents. However, when facing images that are significantly different in content, context, or style, the effectiveness of multi-modal large models, including GPT-4V, will decrease. In these cases, it can be difficult for the large model to establish meaningful semantic associations between images and generate accurate answers.

[0003] Based on the above analysis, this defect can be attributed to the lack of semantic alignment in existing multi-modal large language models when processing multiple images. Specifically, the current multi-modal large language models when processing multiple images will include two stages:

[0004] Independent perception: This step uses a visual perception component (such as Q-Former, linear layers, and Resampler) to map the visual features encoded by the pre-trained visual backbone network into visual tokens in the feature space of the large language model. In this way, the image visual tokens received by the large model will contain its visual features.

[0005] Comprehensive analysis: After obtaining the visual tokens of each image, the model uses a cross-modal attention mechanism to fuse them with the text tokens of the instruction. This step allows the model to combine information from different modalities and images.

[0006] In this way, the multi-modal large model mainly handles multi-image reasoning in the comprehensive analysis stage, which leads to the problem of semantic misalignment, hindering it from exploring the connections between images. This is because the visual tokens used in the comprehensive analysis step may lack the information needed to "link" multiple images for identifying connections, which is particularly significant when the input images present large differences. Specifically, the visual tokens are affected by the inductive bias of the training data (such as image-caption pairs), leading them to selectively express prominent visual content. Therefore, in complex contextual environments, the focus of the visual tokens of different images may be different, hindering semantic alignment. In addition, in the comprehensive analysis stage, the large model performs attention computation on the visual tokens and text tokens together, which leads to the interaction between visual tokens being overshadowed in a large number of token interactions, making it difficult to perform semantic alignment.

[0007] The method for solving the above semantic misalignment problem is to introduce context semantics from context images as guidance in the perception stage. By utilizing context images as references, the visual markers of the current perception image can be accurately aligned with the context semantics before the comprehensive analysis step. However, it should be noted that due to the presence of irrelevant or even noise information, all context images in the input multi-modal instruction may not be directly related to the current perception image. In this case, directly incorporating all the information of all context images may bring challenges to the perception process. Therefore, in addition to the above semantic misalignment problem, how to accurately extract context semantics highly related to the current perception image is also an important problem worthy of careful consideration. SUMMARY

[0008] The purpose of the present application is to solve the problems existing in the prior art and provide a semantic alignment method and system in a multi-image multi-modal large language model.

[0009] In order to achieve the above-mentioned purpose of the application, the technical scheme is as follows:

[0010] In a first aspect, the present application provides a semantic alignment method in a multi-image multi-modal large language model, comprising the following steps:

[0011] S1. Constructing multi-image multi-modal samples from existing single-image data sets, each multi-image multi-modal sample containing a plurality of final images and a final description text corresponding to each multi-image multi-modal sample; wherein each final image contains the same object, and the same object in each final image is presented in different poses, lighting conditions or viewing angles;

[0012] S2. Obtaining a trained semantic alignment model, inputting the prompt words containing the task to be completed and the picture to be aligned into the trained semantic alignment model, and outputting the answer corresponding to the prompt words of the task to be completed by the semantic alignment model;

[0013] The semantic alignment model comprises a visual encoder, a bidirectional guidance module and a large language model, the bidirectional guidance module comprises two Q-former modules and an adaptive weight adjustment module, the first Q-former module and the second Q-former module are interacted through a linear layer, and the first Q-former module and the second Q-former module share parameters;

[0014] The semantic alignment model is trained on a multi-graph multi-modal sample generated multi-graph multi-modal data set, and in the training process of the semantic alignment model, the parameters of the visual encoder, the first Q-former module, the second Q-former module and the large language model are fixed and not updated, and only the adaptive weight adjustment module and the linear layer parameters for interaction are updated.

[0015] On the basis of the above scheme, each step can be implemented in the following preferred specific manner.

[0016] As a preferred embodiment of the first aspect, the specific process of step S1 is as follows:

[0017] S11. Extract pictures of different character roles from the single-picture data set of the character and construct several picture groups, each picture group consisting of two character role pictures, wherein one character role picture in one picture group contains the same character role as one character role picture in another picture group, and the other character role picture in one picture group contains a different character role from the other character role picture in another picture group, and the pictures containing the same character role are pictures of the same character role at different angles;

[0018] S12. Segment the pictures in each picture group using Segment Anything, and obtain a segmented picture group for each picture group;

[0019] S13. Preprocess each segmented picture group, and obtain a preprocessed picture group for each segmented picture group. Place the two pictures in each preprocessed picture group on the same background picture, and obtain a mask picture for each preprocessed picture group. Each pixel value on the background picture is 0.

[0020] S14. Randomly select one of the diversified picture background descriptions generated from the large language model as the picture background description for each mask picture, and input the mask picture and its corresponding picture background description into the Stable-Diffusion-Inpainting model to complete the filling process. Each mask picture corresponds to a final picture that meets the picture background description and has a harmonious overall picture.

[0021] S15. Use the large language model to describe the same character role in each picture group obtained in S11 to obtain an initial description. Input the prompt words for guiding the large language model to optimize the initial description into the large language model, and optimize the initial description by the large language model to obtain a final description text.

[0022] As a preferred embodiment of the first aspect, in step S13, when pre-processing each segmented picture group, each segmented picture group contains two segmented character pictures, each segmented character picture is scaled in height to a preset interval range and is scaled in width proportionally, and each segmented character picture corresponds to a scaled character picture, and the pre-processing picture group is generated from the scaled character picture.

[0023] As a preferred embodiment of the first aspect, in step S15, the large language model used to describe the character role adopts InstructBLIP, and the large language model used to optimize the initial description adopts ChatGPT.

[0024] As a preferred embodiment of the first aspect, in the semantic alignment model, the specific processing flow is as follows:

[0025] S21. Input the picture to be semantically aligned into a visual encoder for encoding to obtain a visual encoded picture;

[0026] S22. After encoding the prompt word to be completed, a text label is obtained, the pre-trained query vector and the text label are spliced to obtain an input query vector;

[0027] S23. When the semantic alignment model processes the i-th visual encoded picture, the i-th visual encoded picture and the input query vector are input into the first Q-former module, and the query vector output by the first Q-former module at the (l-1) layer is input into the first linear layer to obtain the semantic guiding feature of the current perception picture;

[0028] S24. The remaining visual encoded pictures except the i-th visual encoded picture are input into the adaptive weight adjustment module, the adaptive weight adjustment module generates normalized adaptive weights according to the remaining visual encoded pictures, the remaining visual encoded pictures and the corresponding adaptive weights are weighted to obtain weighted visual encoded pictures, and finally the weighted visual encoded pictures are summed to obtain a fused visual encoded picture;

[0029] S25. The fused visual encoded picture and the input query vector are input into the second Q-former module, and the query vector output by the second Q-former module at the (l-1) layer is added to the semantic guiding feature of the current perception picture to obtain an updated output query vector;

[0030] S26. The remaining layers of the second Q-former module extract relevant details from the fused visual encoded picture, and the query vector output by the k>l layer of the second Q-former module is input into the second linear layer to obtain a context semantic feature;

[0031] S27. Adding the query vector and the context semantic feature output by the first Q-former module at the (k-1)th layer as the input of the first Q-former module at the kth layer; and outputting the final visual label of the current perception picture with enhanced semantic alignment by the kth layer and the remaining layers of the first Q-former module;

[0032] S28. Processing each visual encoding picture by the semantic alignment model until all the visual encoding pictures are processed, inputting the final visual label and the text label corresponding to each visual encoding picture into the large language model, and outputting the final answer by the large language model.

[0033] As a preferred embodiment of the first aspect, the adaptive weight adjustment module generates normalized adaptive weights The specific implementation is as follows:

[0034]

[0035] wherein Linear represents a linear layer, I m,j represents the remaining one visual encoding picture except the ith visual encoding picture; and softmax represents a softmax function.

[0036] In a second aspect, the present application provides a semantic alignment system in a multi-picture multi-modal large language model, comprising:

[0037] A data acquisition module is configured to construct multi-picture multi-modal samples from an existing single-picture data set, each multi-picture multi-modal sample containing a plurality of final pictures and a final description text corresponding to each multi-picture multi-modal sample; wherein each final picture contains the same object, and the same object in each final picture is presented in different poses, lighting conditions or viewing angles.

[0038] A result acquisition module is configured to acquire a trained semantic alignment model, input a prompt word containing a task to be completed and a picture to be semantically aligned into the trained semantic alignment model, and output an answer corresponding to the prompt word of the task to be completed by the semantic alignment model.

[0039] The semantic alignment model comprises a visual encoder, a bidirectional guiding module and a large language model, the bidirectional guiding module comprises two Q-former modules and an adaptive weight adjustment module, the first Q-former module and the second Q-former module interact through a linear layer, and the first Q-former module and the second Q-former module share parameters.

[0040] The semantic alignment model is trained on a multi-image multi-modal sample generated multi-image multi-modal data set, and in the training process of the semantic alignment model, the parameters of the visual encoder, the first Q-former module, the second Q-former module and the large language model are fixed and not updated, and only the adaptive weight adjustment module and the linear layer parameters for interaction are updated.

[0041] In a third aspect, the present application provides a computer program product, comprising computer programs / instructions which, when executed by a processor, can implement the semantic alignment method in the multi-image multi-modal large language model according to any one of the above first aspect.

[0042] In a fourth aspect, the present application provides a computer readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the semantic alignment method in the multi-image multi-modal large language model according to any one of the above first aspect can be implemented.

[0043] In a fifth aspect, the present application provides a computer electronic device, comprising a memory and a processor;

[0044] The memory is used to store a computer program;

[0045] The processor is used to, when executing the computer program, can implement the semantic alignment method in the multi-image multi-modal large language model according to any one of the above first aspect.

[0046] Compared with the prior art, the present application has the following beneficial effects:

[0047] (1) The present application introduces bidirectional guidance from context pictures to perform semantic alignment between multiple pictures. Experimental results show that performance is improved by 37% and 22% on Group captioning and Storytelling tasks, respectively. Even with a parameter amount of only 7 billion, the performance exceeds that of commercial closed-source large models such as GPT-4V and GeminiPro.

[0048] (2) The method of the present application can not only perform semantic alignment in the case of large differences between pictures, but also has stable performance in the case of high similarity between pictures. On the Change captioning task, the performance still exceeds that of existing open-source multi-modal large models, with a performance improvement of 39%. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 The flowchart of the steps of the present application;

[0050] Figure 2A schematic diagram of the multi-image multi-modal sample constructed for the present application;

[0051] Figure 3 A schematic diagram of the semantic alignment model structure of the present application;

[0052] Figure 4 A system block diagram of the present application. DETAILED DESCRIPTION

[0053] In order to make the above objectives, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the spirit of the present application, so the present application is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present application can be combined accordingly without conflict.

[0054] In the description of the present application, it should be understood that the terms "first", "second" are only used for distinguishing purposes of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can be explicitly or implicitly included at least one of the features.

[0055] The technical problem to be solved by the present application is that the existing multi-modal large model usually adopts a two-stage strategy of "independent perception-comprehensive analysis" when processing image-text instructions. However, in the multi-image scenario, especially when the differences between pictures are significant, the visual markers extracted in the independent perception stage will cause semantic misalignment between different pictures without the guidance of context pictures. Such semantically misaligned visual markers will greatly mislead the subsequent comprehensive analysis stage, so that the large model cannot correctly understand the correlation between pictures, and thus generate inaccurate answers.

[0056] In a preferred embodiment of the present application, as shown in Figure 1 In a preferred implementation of the present application, the semantic alignment method in the above multi-image multi-modal large language model includes the following S1-S2 steps. The specific implementation process will be described below.

[0057] The present application mainly includes two parts, one is the generation of multi-image multi-modal samples based on picture filling technology, and the other is the semantic alignment based on the bidirectional guidance of context pictures.

[0058] S1. Constructing multi-graph multi-modal samples from an existing single-graph dataset, each multi-graph multi-modal sample containing a plurality of final pictures and a final description text corresponding to each multi-graph multi-modal sample; wherein each final picture contains the same object, and the same object in each final picture is presented in different poses, lighting conditions or perspectives.

[0059] It should be noted that the step S1 of the present application is a multi-graph multi-modal sample generation process based on image filling technology. Most existing multi-graph data sets for fine-tuning multi-modal large language models may exhibit significant similarity between images of multi-modal instructions, lacking associated pictures with significant differences (e.g., different contexts, different styles). For this purpose, the present application constructs a new large-scale multi-modal data set named MmLINK based on image filling technology for multi-modal large model research. The data set contains 69,000 multi-graph multi-modal samples, which are specifically used to enhance the model's cross-modal multi-image semantic alignment and association mining capabilities. Each multi-graph multi-modal sample contains two final pictures I1, I2 and a final description text T aligning the common points, and each final picture corresponds to two final pictures containing the same object, but presented in different poses, lighting conditions or perspectives. The object in the present application refers to the associated semantics contained in the picture.

[0060] It is worth noting that in order to ensure the diversity of objects in the multi-graph multi-modal samples in the MmLINK data set, the present application collects pictures containing various objects from different picture sources, covering people, furniture items, icons and book covers. For different types of objects, the present application uses slightly different processes to create corresponding samples.

[0061] In the present embodiment, as shown in Figure 2 , taking "people" as an example, the multi-graph multi-modal sample construction process is explained. The generation process of the multi-graph multi-modal sample includes two stages: mask picture generation and background filling. Specifically as follows:

[0062] 1) Mask picture generation

[0063] S11. Extracting pictures of different character roles from the single-graph dataset of people and constructing a plurality of picture groups, each picture group consisting of two character role pictures, wherein the character role contained in one character role picture in a picture group is the same as the character role contained in one character role picture in another picture group, and the character role contained in the other character role picture in a picture group is different from the character role contained in the other character role picture in another picture group. The pictures containing the same character role are pictures of the same character role at different angles.

[0064] In the present embodiment, as shown in Figure 2As shown, images of different characters are extracted from the DeepFashion dataset to construct two image groups. The first image group [C1, C2] and the second image group [C1′, C3] contain the same character. C1 and C1′ are two images of the same character from different angles, while C2 and C3 are images of two other different characters. In this way, the two image groups contain the same character but are presented in different poses, lighting conditions, or perspectives. Furthermore, each image group has an additional character as a distractor to increase the difficulty of the task. Subsequent steps then use these images of characters to form two mask images.

[0065] S12. Use Segment Anything to segment the images in each image group, resulting in a segmented image group for each image group.

[0066] In this embodiment, as Figure 2 As shown, since the character images in the DeepFashion dataset contain background noise, it is necessary to segment the four character images in the two image groups to remove the background noise. Specifically, Segment Anything is used to segment the four character images (C1, C1′, C2, C3) obtained in step S11. Because the backgrounds of the images in the DeepFashion dataset are all light-colored, making the characters stand out, the segmentation result with the highest confidence among the many outputs of Segment Anything is selected as the segmented character image. In this embodiment, corresponding to the four character images obtained in the above steps, a total of four segmented character images are obtained after segmentation, and these four segmented character images are restored to the segmented image groups according to the original image group combination method. Corresponding to the two image groups in the above process, after segmentation, the first segmented image group is... The second group consists of the segmented image groups. The image of the person obtained after segmenting C1. The image of the person obtained after C2 is segmented. The image of the person obtained after segmentation of C1′. The image of the person is obtained after C3 is segmented.

[0067] S13. Preprocess each group of segmented images. Each group of segmented images corresponds to a preprocessed image group. Place two images from each preprocessed image group on the same background image. Each preprocessed image group corresponds to a mask image. The pixel value of each pixel in the background image is 0.

[0068] It should be noted that in step S13 of the present application, when preprocessing each group of segmented picture groups, each group of segmented picture groups contains two segmented character pictures, the height of each segmented character picture is scaled to a preset interval range and the width is scaled proportionally, and each segmented character picture corresponds to a scaled character picture, and the preprocessed picture group is generated from the scaled character picture.

[0069] In the present embodiment, as shown in Figure 2 , considering that the character picture is usually "thin and tall", that is, the height is much larger than the width, the height of each segmented character picture is scaled to a random number in the interval of 200 to 250, and the width is scaled proportionally, so as to obtain a scaled character picture, which can ensure that the character picture can be completely placed on the black background picture. Then, the two scaled character pictures of the same group are placed on a background picture with a size of 512x512, thus forming a mask picture. Corresponding to the two picture groups in the above process, two mask pictures I1, I2 are obtained after the mask generation process.

[0070] 2) Background filling

[0071] In the background filling process, the mask picture obtained in the previous step is filled with various background pixels to become a complete final picture, and a corresponding final description text is generated for each picture group.

[0072] S14. Randomly select one of the diversified picture background descriptions generated from the large language model as the picture background description of each mask picture, and input the mask picture and the corresponding picture background description into the Stable-Diffusion-Inpainting model to complete the filling process. Each mask picture corresponds to a final picture that conforms to the picture background description and has a harmonious overall picture.

[0073] In the present embodiment, as shown in Figure 2 , first, request ChatGPT to obtain 500 diversified picture background descriptions for filling the mask picture. For each mask picture, a picture background description is randomly selected, and the Stable-Diffusion-Inpainting model is used to complete the filling process. The model receives the mask picture and the picture background description as input, and fills the mask picture with relatively harmonious background pixels that conform to the image background description. In this way, the mask picture is filled into a final picture. Corresponding to the two mask pictures in the above process, two final pictures I1, I2 are obtained after the mask generation process.

[0074] S15. Using a large language model to describe the same character roles in each group of picture groups obtained in S11 to obtain an initial description; input the prompt word for guiding the large language model to optimize the initial description to the large language model, and optimize the initial description by the large language model to obtain the final description text T.

[0075] In S15 of the present application, the large language model used to describe the character roles and the large language model used to optimize the initial description can be the same or different. In the present embodiment, as shown in Figure 2 the InstructBLIP model is first used to describe the same character roles in each group of picture groups in the above steps to obtain an initial description, and then ChatGPT is used to optimize the initial description to obtain the final description text T. The initial description aims to describe the similarities of two different angle pictures (such as C1, C1') of the same character role, i.e. to describe the character role.

[0076] S2. Obtain a trained semantic alignment model, input the prompt word containing the task to be completed and the picture to be semantically aligned into the trained semantic alignment model, and output the answer corresponding to the prompt word of the task to be completed by the semantic alignment model;

[0077] The semantic alignment model comprises a visual encoder, a bidirectional guide module and a large language model, the bidirectional guide module comprises two Q-former modules and an adaptive weight adjustment module, the first Q-former module and the second Q-former module interact through a linear layer, and the first Q-former module and the second Q-former module share parameters;

[0078] The semantic alignment model is trained on a multi-picture multi-modal dataset generated by a multi-picture multi-modal sample, and in the training process of the semantic alignment model, the parameters of the visual encoder, the first Q-former module, the second Q-former module and the large language model are fixed and not updated, and only the parameters of the adaptive weight adjustment module and the linear layer for interaction are updated.

[0079] It should be noted that the step S2 of the present application is a semantic alignment process based on context picture bidirectional guidance. Visual markers are extracted for the current perception picture and the context picture respectively, and mutual guidance is performed in the extraction process of the visual markers. Specifically, in order to solve the semantic alignment problem of the existing multi-modal large model when processing images with different contexts, the present application introduces a bidirectional semantic guidance mechanism in the image perception stage, generates visual markers with enhanced semantic alignment from patch-level features, and constructs a semantic alignment model based on the visual markers. In the comprehensive analysis stage, the visual markers and the input text markers are processed by the large language model of the semantic alignment model to generate the final prediction result. The bidirectional semantic guidance mechanism mainly includes two interactive processes: auxiliary visual marker extraction and context semantic generation. Through the bidirectional semantic guidance mechanism based on the context picture, the semantic alignment of the semantic alignment model for multiple pictures can be promoted. The overall process of the semantic alignment model can be divided into four steps.

[0080] A) Image feature generation

[0081] S21. The picture to be subjected to semantic alignment is input to a visual encoder for encoding to obtain a visual encoding picture.

[0082] It should be noted that in the step S21 of the present application, as shown in Figure 3 , given N input pictures, the input pictures are converted into patch-level features by using a visual encoder to obtain visual encoding pictures. The i-th input picture is taken as a current perception picture, and the visual encoding picture of the current perception picture is denoted as i . i,j} P , where P is the number of patches, and j represents the patch index. In the present embodiment, the visual encoder adopts an EVA-CLIP model.

[0083] S22. After encoding the prompt word to be completed, a text marker is obtained, and a pre-trained query vector and the text marker are spliced to obtain an input query vector.

[0084] It should be noted that in the step S22 of the present application, as shown in Figure 3 , the text marker is Figure 3 in the green marked circle, and the pre-trained query vector is Figure 3 in the blue marked circle.

[0085] B) Auxiliary visual marker extraction

[0086] Firstly, in the auxiliary visual marker extraction process, the semantic alignment model applies a first Q-former module to process the current perception picture and extract an initial visual marker, i.e. the semantic guidance feature h iThen, in the context semantic generation process, the application utilizes another Q-former module as a new visual marker generator. Under the guidance of the initial visual marker h i , the second Q-former module extracts the synchronous context semantics from the context picture to obtain the context semantic feature c i . Finally, the context semantics are transmitted back to the first Q-former module to guide the update of the visual marker based on the natural language query, and the semantic alignment of the visual marker is enhanced by aligning the context information.

[0087] The specific steps corresponding to the auxiliary visual marker extraction process are as follows:

[0088] S23. When the semantic alignment model processes the i-th visual encoding picture, the i-th visual encoding picture and the input query vector are input into the first Q-former module, and the query vector output by the first Q-former module at the (l-1)-th layer is input into the first linear layer to obtain the semantic guidance feature h i of the current perception picture.

[0089] It should be noted that the step S23 of the application is intended to provide guidance from the current perception picture to the context picture, so that the subsequent steps can extract information highly related to the current perception picture from the context picture for guidance. Specifically, for the visual encoding picture I i , the application uses the Q-former module proposed in InstructBLIP as the first Q-former module to perform semantic compression and extract visual markers (the semantic guidance feature h i of the current perception picture). The first Q-former module is stacked by several cross-attention layers, which initially takes the pre-trained query vector and the text marker t as Query, and the visual encoding picture I i as the Key and Value of cross-attention, and updates the output query vector through each cross-attention layer. It can be represented as:

[0090]

[0091] where Q l-1 is the (l-1)-th layer of the first Q-former module; represents the output query vector of the (l-1)-th layer of the first Q-former module (that is, the input query vector of the l-th layer of the first Q-former module); represents the output query vector of the (l-2)-th layer of the first Q-former module (that is, the input query vector of the (l-1)-th layer of the first Q-former module).

[0092] In order to extract the visual mark of the current perception picture for subsequent guiding process, the output query vector of the first Q-former module at the (l-1)th layer is obtained After passing through the linear layer, it is used as the semantic guiding feature of the current perception picture:

[0093]

[0094] C) Contextual semantic generation

[0095] S24. Input the remaining visual encoding pictures except the ith visual encoding picture into the adaptive weight adjustment module, generate normalized adaptive weights according to the remaining visual encoding pictures by the adaptive weight adjustment module, weight the remaining visual encoding pictures with the respective corresponding adaptive weights, obtain the weighted visual encoding pictures, and finally sum the weighted visual encoding pictures to obtain the fused visual encoding picture.

[0096] It should be noted that in step S24 of the present application, the specific implementation mode of the adaptive weight adjustment module generating normalized adaptive weights is as follows:

[0097]

[0098] Wherein, Linear represents a linear layer, I m,j represents the remaining one visual encoding picture except the ith visual encoding picture; softmax represents a softmax function.

[0099] In the present embodiment, as Figure 3 shown, after obtaining the semantic guiding feature of the current perception picture, it is needed to use it to guide the contextual pictures, so as to filter out the redundant information in the contextual pictures and extract the information in the contextual pictures highly related to the current perception picture. Specifically, the pictures other than the current perception picture are used as the contextual pictures, and the set of the contextual pictures is represented as {I m} m≠i The patch-level feature (visual encoding picture) corresponding to the mth (m≠i) contextual picture is represented as The adaptive weight adjustment module is used to assign weights to each contextual picture patch, so as to weaken the influence of irrelevant patches. The normalized patch-level weights are generated through a linear layer and a softmax function, then the visual encoding pictures of each contextual picture are reweighted, and the weighted visual encoding pictures are summed to obtain the fused visual encoding picture This process reduces the influence of irrelevant patches and enhances the fused visual encoding picture Patch-level alignment between the current perception image and the input query vector.

[0100] S25. input the fusion visual encoding image and the input query vector into the second Q-former module, and input the query vector w l the semantic guidance feature h i and add, to obtain the updated output query vector

[0101] It should be noted that in step S25 of the present application, the auxiliary visual marker extraction process needs the context semantic features of the context image to guide the visual marker extraction of the current perception image. However, it should be noted that all the context images in the input may not be directly related to the current perception image and the natural language query. In addition, even if the image is highly relevant, it may also contain irrelevant or noise information. In order to solve these problems, the present application reuses the structure of the Q-former module in the context semantic generation process, and constructs a second Q-former module to extract semantics from the fusion visual encoding image. The second Q-former module has the same structure as the first Q-former module and the same parameters. The second Q-former module also takes the pre-trained query vector and the text marker t as Query, and is different from the first Q-former module in that the second Q-former module takes the fusion visual encoding image as the Key and Value of cross-attention, and the process of extracting semantics is as follows:

[0102]

[0103] wherein, W l-1 is the (l-1)th layer of the second Q-former module; w l represents the output query vector of the (l-1)th layer of the second Q-former module (that is, the query vector input into the lth layer of the second Q-former module); w l-1 represents the output query vector of the (l-2)th layer of the second Q-former module (that is, the query vector input into the (l-1)th layer of the second Q-former module).

[0104] In the lth layer of the second Q-former module, the obtained semantic guidance feature h i is added, to obtain the updated output query vector

[0105]

[0106] S26. The remaining layers of the second Q-former module extract relevant details from the fused visual encoding map The query vector w k+1 is outputted by the kth layer of the second Q-former module. i .

[0107] It is to be noted that in step S26, the remaining layers of the second Q-former module can efficiently extract relevant details from the fused visual encoding map under the guidance of the semantic guidance feature h i of the current perception image, which helps the second Q-former module to generate context semantic features highly relevant to the current perception image by effectively reducing the redundant details of the context image in the multi-modal instruction. Specifically, the context semantic feature c k+1 is obtained by extracting the context semantic from the query vector w i outputted by the kth layer of the second Q-former module through a second linear layer.

[0108] c i = Linear(w k+1 )

[0109] The generated context semantic feature gathers information related to the current perception image in the entire context, which promotes the generation of semantically aligned visual markers.

[0110] D) Final visual marker generation

[0111] S27. The query vector outputted by the (k-1)th layer of the first Q-former module is added to the context semantic feature c i as the input of the kth layer of the first Q-former module The final visual marker of the current perception image with enhanced semantic alignment is generated by the output of the kth layer and the remaining layers of the first Q-former module.

[0112] It is to be noted that in step S27, the context semantic feature is used to guide the process of generating the final visual marker of the current perception image, thereby promoting the semantic alignment between the current perception image and the context image. Specifically, after obtaining the context semantic feature, the kth layer of the first Q-former module is selected, and the context semantic feature c i is added to the input of the kth layer of the first Q-former module to guide:

[0113]

[0114] The subsequent layers of the first Q-former module will generate the final visual tokens of the current perception picture with enhanced semantic alignment, with reference to the contextual semantics. These final visual tokens with enhanced semantic alignment will be delivered to the large language model, together with the text tokens, to generate the final answer.

[0115] S28. Process each visual encoding picture by the semantic alignment model until all visual encoding pictures are processed, and input the final visual tokens and text tokens corresponding to each visual encoding picture into the large language model, and output the final answer by the large language model.

[0116] It should be noted that in the present application, the layer indexes k and l can be selected according to actual conditions to interact with a certain layer in the first Q-former module and the second Q-former module, as long as k≥l. In the embodiment of the present application, k and l are set to the same value, both representing the last layer of the first Q-former module and the second Q-former module, that is, the two interactions are performed at the final layer of the first Q-former module and the second Q-former module, l=k=12.

[0117] In addition, it should be noted that in the present application, the parameters of the visual encoder and the large language model are frozen for efficient training. In addition, the first Q-former module and the second Q-former module are also frozen. This means that only the two linear layers and the adaptive weight adjustment module for the interaction of the first Q-former module and the second Q-former module need to be fine-tuned, with a total of 4.3 million (0.05%) trainable parameters. The present application uses the AdamW optimizer to train the semantic alignment model, and sets the parameters to β=(0.9, 0.999); uses a learning rate of 2e-5 and a batch size of 20 for training.

[0118] In order to better demonstrate the specific implementation and technical effects of the present application, the semantic alignment method in the multi-image multi-modal large language model shown in steps S1-S2 in the above preferred implementation will be applied to a specific example.

[0119] Embodiment

[0120] The specific implementation process of the semantic alignment method in the multi-image multi-modal large language model used in this embodiment is as described above and will not be repeated here.

[0121] To demonstrate the technical effects of the embodiment, the present application is verified on Group Captioning and Storytelling tasks. Among them, the Group Captioning task requires the model to find the common characteristics of a group of pictures and describe them in natural language, and the present application is verified on three public datasets, namely Conceptual, Animal and Vehicle. The Storytelling task requires the model to continue the story according to the content of the last picture given a number of pictures and the corresponding story plot, and the present application is verified on three public datasets, namely AESOP, VIST and DM800K.

[0122] To objectively evaluate the performance of the method of the embodiment, the performance indicators on the above two tasks and six datasets all use the commonly used indicators ROUGE-L, CIDEr and BLEU-4. The obtained experimental results are shown in Tables 1 and 2. In Tables 1 and 2, R represents ROUGE-L, C represents CIDEr, and B represents BLEU-4.

[0123] Table 1 Results of the present application on the Group Captioning task

[0124]

[0125] Table 2 Results of the present application on the Storytelling task

[0126]

[0127] From the above results, it can be seen that in all tasks, all datasets and all indicators, the present application exceeds the open-source multi-modal large model and the closed-source commercial multi-modal large model, including the current strongest multi-modal large model such as GPT-4V and Gemini Pro. Specifically, in terms of the CIDEr indicator, the present application achieves a performance improvement of 37% on the Group Captioning task and an improvement of 22% on the Storytelling task compared to the current strongest model. These demonstrate the excellent multi-image semantic alignment capability of the method of the present application in the scenario where the differences between pictures are large.

[0128] To verify that the method of the present application still retains certain semantic alignment capabilities in the scene where the pictures are more similar, the present example conducts experiments on the Change Captioning task. This task requires the model to find the differences in two highly similar pictures and describe them in natural language. The present application tests on four public datasets, IEdit, Spot-the-Diff, Birds-to-Words, and CLEVR-Change. The experimental results are shown in Table 3. In Table 3, R represents ROUGE-L, C represents CIDEr, and B represents BLEU-4.

[0129] Table 3 Results of the present application on the Change Captioning task

[0130]

[0131] As can be seen from Table 3, the average indicators of the present example on the Change Captioning task all exceed the open source multi-modal large model, and the performance is improved by 39% on the CIDEr indicator. This shows that the method of the present application can still retain certain semantic alignment capabilities in the scene where the pictures are more similar.

[0132] In addition, it should be noted that the semantic alignment method in the multi-image multi-modal large language model in the above embodiments can be essentially executed by a computer program or module. Therefore, based on the same inventive concept, another preferred embodiment of the present application also provides a semantic alignment system in a multi-image multi-modal large language model corresponding to the semantic alignment method in the multi-image multi-modal large language model provided by the above embodiments, as shown in Figure 4 which includes:

[0133] A data acquisition module for constructing multi-image multi-modal samples from existing single-image datasets, each multi-image multi-modal sample containing a plurality of final images and a final description text corresponding to each multi-image multi-modal sample; wherein each final image contains the same object, and the same object in each final image is presented in different poses, lighting conditions or perspectives;

[0134] A result acquisition module for acquiring a trained semantic alignment model, inputting the prompt words of the task to be completed and the pictures to be semantically aligned into the trained semantic alignment model, and outputting the answers corresponding to the prompt words of the task to be completed by the semantic alignment model;

[0135] The semantic alignment model comprises a visual encoder, a bidirectional guiding module, and a large language model, the bidirectional guiding module comprises two Q-former modules and an adaptive weight adjustment module, the first Q-former module and the second Q-former module interact through a linear layer, and the first Q-former module and the second Q-former module share parameters;

[0136] The semantic alignment model is trained on a multi-graph multi-modal dataset generated by a multi-graph multi-modal sample, and in the training process of the semantic alignment model, the parameters of the visual encoder, the first Q-former module, the second Q-former module, and the large language model are fixed and not updated, and only the parameters of the adaptive weight adjustment module and the linear layer used for interaction are updated.

[0137] It can be understood that the semantic alignment method in the multi-graph multi-modal large language model described in S1-S2 can be realized by a computer program. Therefore, based on the same inventive concept, another preferred embodiment of the present application also provides a computer program product corresponding to the semantic alignment method in the multi-graph multi-modal large language model provided by the above-mentioned embodiment, which includes a computer program / instruction, which can realize the semantic alignment method in the multi-graph multi-modal large language model as described in the above-mentioned embodiment when executed by a processor.

[0138] Similarly, based on the same inventive concept, another preferred embodiment of the present application also provides a computer electronic device corresponding to the semantic alignment method in the multi-graph multi-modal large language model provided by the above-mentioned embodiment, which includes a memory and a processor;

[0139] The memory is used to store a computer program;

[0140] The processor is used to realize the semantic alignment method in the multi-graph multi-modal large language model in the above-mentioned embodiment when executing the computer program.

[0141] In addition, the logical instructions in the above-mentioned memory can be realized in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the embodiments of the present application.

[0142] Thus, based on the same inventive concept, another preferred embodiment of the present application also provides a computer readable storage medium corresponding to the semantic alignment method in the multi-graph multi-modal large language model provided by the above-mentioned embodiments. The storage medium stores a computer program, and when the computer program is executed by a processor, the semantic alignment method in the multi-graph multi-modal large language model in the above-mentioned embodiments can be implemented.

[0143] It can be understood that the above-mentioned storage medium can include a random access memory (RAM) and can also include a non-volatile memory (NVM), such as at least one disk memory. Meanwhile, the storage medium can also be a U disk, a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0144] It can be understood that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0145] In addition, it should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-mentioned system can refer to the corresponding process in the above-mentioned method embodiments, which will not be described here. In the embodiments provided in the present application, the division of steps or modules in the system and method is only a logical functional division, and actual implementation can have another division mode, for example, multiple modules or steps can be combined or integrated together, or a module or step can be split.

[0146] The above-mentioned embodiments are only a preferred scheme of the present application, and are not intended to limit the present application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application. Therefore, any technical scheme obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present application.

Claims

1. A semantic alignment method in a multi-graph, multimodal large language model, characterized in that, include: S1. Construct multi-image multimodal samples from existing single-image datasets. Each multi-image multimodal sample contains several final images and corresponding final description text. Each final image contains the same object, and the same object in each final image is presented in different poses, lighting conditions, or viewpoints. S2. Obtain a trained semantic alignment model. Input the prompt words containing the task to be completed and the image to be semantically aligned into the trained semantic alignment model. The semantic alignment model will output the answer corresponding to the prompt words of the task to be completed. The semantic alignment model includes a visual encoder, a bidirectional guidance module, and a large language model. The bidirectional guidance module includes two Q-former modules and an adaptive weight adjustment module. The two Q-former modules interact through a linear layer and share parameters. The semantic alignment model is trained on a multi-graph multimodal dataset generated from multi-graph multimodal samples. During the training of the semantic alignment model, the parameters of the visual encoder, the first Q-former module, the second Q-former module, and the large language model are fixed and not updated. Only the parameters of the adaptive weight adjustment module and the linear layer used for interaction are updated. The processing flow of the semantic alignment model is as follows: S21. Input the image to be semantically aligned into the visual encoder for encoding to obtain the visual encoded image; S22. After encoding the prompts for completing the task, text tags are obtained. The pre-trained query vector and the text tags are concatenated to obtain the input query vector. S23. When the semantic alignment model processes the i-th visual encoding image, the i-th visual encoding image and the input query vector are input together into the first Q-former module. The query vector output by the (l-1)-th layer of the first Q-former module is passed through the first linear layer to obtain the semantic guidance features of the current perceived image. S24. Input the remaining visual coding maps, excluding the i-th visual coding map, into the adaptive weight adjustment module. First, the adaptive weight adjustment module generates normalized adaptive weights based on the remaining visual coding maps. Then, the remaining visual coding maps are weighted with their respective adaptive weights to obtain the weighted visual coding maps. The summation of the weighted visual coding maps is then used to obtain the fused visual coding map. S25. Input the fused visual encoding map and the input query vector into the second Q-former module, add the query vector output by the (l-1)th layer of the second Q-former module to the semantic guidance features of the current perceived image, and obtain the updated output query vector; S26. The remaining layers of the second Q-former module extract relevant details from the fused visual coding graph, and after passing the query vector output by the k≥l layer of the second Q-former module through the second linear layer, the context semantic features are obtained. S27. The query vector output from the (k-1)th layer of the first Q-former module and the contextual semantic features are added together as the input to the kth layer of the first Q-former module; the final visual label of the current perceived image with enhanced semantic alignment is generated from the output of the kth layer and the remaining layers of the first Q-former module. S28. The semantic alignment model processes each visual encoding map until all visual encoding maps have been processed. The final visual tag and text tag corresponding to each visual encoding map are then input into the large language model, which outputs the final answer.

2. The semantic alignment method in a multi-graph, multimodal large language model as described in claim 1, characterized in that, The specific process of step S1 is as follows: S11. Extract images of different characters from the single-image dataset and construct several image groups. Each image group consists of two character images. In one image group, the character in one character image is the same as the character in another image group. In another image group, the character in the other character image is different from the character in another image group. Images containing the same character are images of the same character from different angles. S12. Use Segment Anything to segment the images in each image group, resulting in a segmented image group for each image group. S13. Preprocess each group of segmented images. Each group of segmented images corresponds to a preprocessed image group. Place two images from each preprocessed image group on the same background image. Each preprocessed image group corresponds to a mask image. The pixel value of each pixel on the background image is 0. S14. Randomly select one of the diverse image background descriptions generated from the large language model as the image background description for each mask image. Input the mask images and their corresponding image background descriptions into the Stable-Diffusion-Inpainting model to complete the filling process. Each mask image corresponds to a final image that conforms to the image background description and has a harmonious overall picture. S15. Use the large language model to describe the same characters in each group of pictures obtained in S11 to obtain the initial description; input the prompt words used to guide the large language model to optimize the initial description into the large language model, and the large language model optimizes the initial description to obtain the final description text.

3. The semantic alignment method in a multi-graph, multimodal large language model as described in claim 2, characterized in that, In step S13, when preprocessing each group of segmented images, each group of segmented images contains two segmented images of people. The height of each segmented image of people is scaled to a preset range and the width is scaled proportionally. Each segmented image of people corresponds to a scaled image of people, and the scaled image of people is used to generate a preprocessed image group.

4. The semantic alignment method in a multi-graph, multimodal large language model as described in claim 2, characterized in that, In step S15, the large language model used to describe the characters is InstructBLIP, and the large language model used to optimize the initial description is ChatGPT.

5. The semantic alignment method in a multi-graph, multimodal large language model as described in claim 1, characterized in that, The adaptive weight adjustment module generates normalized adaptive weights. The specific implementation method is as follows: Where Linear represents a linear layer, I m,j It represents the remaining visual encoding map excluding the i-th visual encoding map; softmax represents the softmax function.

6. A semantic alignment system in a multi-graph, multimodal large language model, characterized in that, include: The data acquisition module is used to construct multi-image multimodal samples from existing single-image datasets. Each multi-image multimodal sample contains several final images and corresponding final description text. Each final image contains the same object, and the same object in each final image is presented in different poses, lighting conditions, or viewpoints. The result acquisition module is used to acquire a trained semantic alignment model. The input of the prompt words containing the task to be completed and the image to be semantically aligned is input into the trained semantic alignment model, and the semantic alignment model outputs the answer corresponding to the prompt words of the task to be completed. The semantic alignment model includes a visual encoder, a bidirectional guidance module, and a large language model. The bidirectional guidance module includes two Q-former modules and an adaptive weight adjustment module. The two Q-former modules interact through a linear layer and share parameters. The semantic alignment model is trained on a multi-graph multimodal dataset generated from multi-graph multimodal samples. During the training of the semantic alignment model, the parameters of the visual encoder, the first Q-former module, the second Q-former module, and the large language model are fixed and not updated. Only the parameters of the adaptive weight adjustment module and the linear layer used for interaction are updated. The processing flow of the semantic alignment model is as follows: S21. Input the image to be semantically aligned into the visual encoder for encoding to obtain the visual encoded image; S22. After encoding the prompts for completing the task, text tags are obtained. The pre-trained query vector and the text tags are concatenated to obtain the input query vector. S23. When the semantic alignment model processes the i-th visual encoding image, the i-th visual encoding image and the input query vector are input together into the first Q-former module. The query vector output by the (l-1)-th layer of the first Q-former module is passed through the first linear layer to obtain the semantic guidance features of the current perceived image. S24. Input the remaining visual coding maps, excluding the i-th visual coding map, into the adaptive weight adjustment module. First, the adaptive weight adjustment module generates normalized adaptive weights based on the remaining visual coding maps. Then, the remaining visual coding maps are weighted with their respective adaptive weights to obtain the weighted visual coding maps. The summation of the weighted visual coding maps is then used to obtain the fused visual coding map. S25. Input the fused visual encoding map and the input query vector into the second Q-former module, add the query vector output by the (l-1)th layer of the second Q-former module to the semantic guidance features of the current perceived image, and obtain the updated output query vector; S26. The remaining layers of the second Q-former module extract relevant details from the fused visual coding graph, and after passing the query vector output by the k≥l layer of the second Q-former module through the second linear layer, the context semantic features are obtained. S27. The query vector output from the (k-1)th layer of the first Q-former module and the contextual semantic features are added together as the input to the kth layer of the first Q-former module; the final visual label of the current perceived image with enhanced semantic alignment is generated from the output of the kth layer and the remaining layers of the first Q-former module. S28. The semantic alignment model processes each visual encoding map until all visual encoding maps have been processed. The final visual tag and text tag corresponding to each visual encoding map are then input into the large language model, which outputs the final answer.

7. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it can implement the semantic alignment method in the multi-graph multimodal large language model as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the semantic alignment method in a multi-graph, multimodal large language model as described in any one of claims 1 to 5.

9. A computer electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to implement the semantic alignment method in a multi-graph, multimodal large language model as described in any one of claims 1 to 5 when executing the computer program.

Images