Image-text question answering method and system based on multi-modal alignment, electronic equipment and medium

Abstract

The invention discloses an image-text question answering method and system based on multi-modal alignment, electronic equipment and a medium, and the method comprises the steps: obtaining a scene image and a question text corresponding to a current dialogue round in response to a dialogue instruction, and determining an initial weight corresponding to the scene image; obtaining a historical image feature set and a historical text feature set of a preset multi-modal feature alignment model; inputting the scene image and the question text into a feature extraction module for feature extraction, and outputting image features and text features; combining the image features and the initial weights into a historical image feature set to obtain an image coding set, and combining the text features into a historical text feature set to obtain a dialogue coding set; performing semantic alignment on the image coding set and the dialogue coding set through a weight matching module; and generating a reply text according to the target image coding features after semantic alignment. In the embodiment of the invention, the image features and the text data can be effectively combined to generate accurate answers.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, system, electronic device and storage medium for image-text question answering based on multimodal alignment. Background Art

[0002] Visual question answering is a new task that has emerged in recent years with the rapid development of natural language processing and computer vision technology. It requires not only a deep understanding of the text, but also the accurate extraction of various attributes of the target from the image. The core of visual question answering lies in the effective interaction between visual features and text features. At present, the more advanced models have mainly innovated in the fusion module. With the application of large language models such as CLIP (Contrastive Language–Image Pre-training), ViT (Vision Transformer), DALL-E, etc. in multi-round continuous dialogues, continuous question answering for visual content has also become feasible.

[0003] The current large language models can understand and generate text descriptions of image content, and perform in-depth reasoning on image content based on questions. However, in multimodal visual question answering, although the current large language models perform well when processing single-modal data, when processing complex interactive picture-text question answering tasks, they often suffer from performance degradation due to image quality issues and insufficient inter-modal fusion, which in turn affects the encoding effect of the language model, and ultimately affects the accuracy and contextual relevance of the question answering. Summary of the invention

[0004] The embodiments of the present application provide a method, system, electronic device and storage medium for image-text question answering based on multimodal alignment, which can effectively combine image features with text data to generate accurate answers.

[0005] In a first aspect, an embodiment of the present application provides a picture-text question-answering method based on multimodal alignment, comprising: In response to the dialogue instruction, a scene image and a question text corresponding to the current dialogue round are obtained, and an initial weight corresponding to the scene image is determined; Obtaining a historical image feature set and a historical text feature set corresponding to a historical dialogue round of a preset multimodal feature alignment model, wherein the multimodal feature alignment model at least includes a feature extraction module and a weight matching module; Inputting the scene image and the question text into the feature extraction module for feature extraction, and outputting image features and text features; Merging the image features and the initial weights into the historical image feature set to obtain an image encoding set, and merging the text features into the historical text feature set to obtain a dialogue encoding set; Semantically aligning the image coding set and the dialogue coding set by the weight matching module to obtain target image coding features; Generate a reply text corresponding to the question text according to the target image encoding feature.

[0006] In some embodiments, the feature extraction module includes an image encoder, a first transformer and a second transformer; inputting the scene image and the question text into the feature extraction module for feature extraction, and outputting image features and text features, comprises: Inputting the scene image into the image encoder so that the image encoder extracts features from the scene image and outputs original features; Performing feature representation on the original feature by the first transformer to obtain image features; The second transformer is used to extract features from the question text to obtain text features.

[0007] In some embodiments, the semantically aligning the image coding set and the dialogue coding set by the weight matching module to obtain target image coding features includes: For each image feature item in the image coding set, weighted combination is performed on the image feature item and the text feature item in the dialogue coding set to obtain an image weight corresponding to the image feature item; The image weight is combined with the image feature corresponding to the image feature item to obtain a target image coding feature.

[0008] In some embodiments, the step of weighting the image feature item and the text feature item in the dialogue encoding set to obtain an image weight corresponding to the image feature item includes: Calculating the similarity value between the image feature item and the text feature item through a preset similarity matrix; Constructing a similarity matrix according to the similarity values ​​of all the image feature items; Semantic weights are assigned to the image feature items based on the similarity matrix to obtain image weights corresponding to the image feature items.

[0009] In some embodiments, the multimodal feature alignment model further includes a large language model; and the step of generating a reply text corresponding to the question text according to the target image encoding feature includes: Performing feature representation on the target image encoding feature by the first transformer to obtain a visual representation feature; The visual representation features and the text features are input into the large language model to predict the answer, and a reply text corresponding to the question text is output.

[0010] In some embodiments, the multimodal feature alignment model further includes a contrastive learning module; after inputting the scene image and the question text into the feature extraction module for feature extraction and outputting image features and text features, the method further includes: The image features and the text features are aligned through the contrastive learning module.

[0011] In some embodiments, the method further comprises: An image loss function is designed by using the image features and the preset training image features, and a text loss function is designed by using the text features and the preset training text features; Setting a sequential consistency loss function according to the image loss function and the text loss function; The contrastive learning module is trained according to the sequential consistency loss function and a preset modality alignment loss function.

[0012] In some embodiments, the multimodal feature alignment model includes an image frame memory module and a continuous dialogue memory module; the step of obtaining a historical image feature set and a historical text feature set corresponding to a historical dialogue round of the preset multimodal feature alignment model includes: Acquire a historical image feature set corresponding to a historical dialogue round through the image frame memory module, and acquire a historical text feature set corresponding to a historical dialogue round through the continuous dialogue memory module; After merging the image features and the initial weights into the historical image feature set to obtain an image encoding set, and merging the text features into the historical text feature set to obtain a dialogue encoding set, the method further includes: The image code set is stored in the image frame memory module, and the dialogue code set is stored in the continuous dialogue memory module.

[0013] In a second aspect, an embodiment of the present application provides a picture-text question-answering system based on multimodal alignment, including: A data acquisition module, used to obtain the scene image and question text corresponding to the current dialogue round in response to the dialogue instruction; A historical data acquisition module, used to acquire a historical image feature set and a historical text feature set corresponding to a historical dialogue round of a preset multimodal feature alignment model, wherein the multimodal feature alignment model at least includes a feature extraction module and a weight matching module; A feature extraction module, used for inputting the scene image and the question text into the feature extraction module for feature extraction, and outputting image features, text features and an initial weight of the scene image; A feature merging module, used for merging the image features and the initial weights into the historical image feature set to obtain an image coding set, and merging the text features into the historical text feature set to obtain a dialogue coding set; A semantic alignment module, used for performing semantic alignment on the image coding set and the dialogue coding set through the weight matching module to obtain target image coding features; A reply generation module is used to generate a reply text corresponding to the question text according to the target image coding features.

[0014] In a third aspect, an embodiment of the present application provides an electronic device comprising at least one processor and a memory for communicating with the at least one processor; the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image-text question and answer method based on multimodal alignment as described in the first aspect.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the image-text question-and-answer method based on multimodal alignment as described in the first aspect.

[0016] The image-text question-and-answer method, device, and storage medium based on multimodal alignment provided in the embodiments of the present application have at least the following beneficial effects: in response to a dialogue instruction, the scene image and question text corresponding to the current dialogue round are obtained, and the initial weight of the scene image in the current dialogue round is determined, and then the historical image feature set and the historical text feature set corresponding to the historical dialogue round of the preset multimodal feature alignment model are obtained, so as to facilitate the subsequent overall analysis of the historical images and historical questions in the dialogue, and then the scene image and question text are input into the feature extraction module for feature extraction, and the image features and text features are output to achieve accurate extraction of key information of the image and key information of the text, thereby improving the accuracy of analyzing the correlation between the problem and the image, and the image features are used as the input of the feature extraction module. The features and initial weights are merged into the historical image feature set to obtain the image coding set, and the text features are merged into the historical text feature set to obtain the dialogue coding set, so that the model can understand and respond to contextual questions more accurately, ensuring that the logical coherence and contextual consistency of the information can be maintained when generating answers. The image coding set and the dialogue coding set are semantically aligned through the weight matching module, and the association weights between images and texts are dynamically adjusted to ensure that in continuous multi-round conversations, each question of the user can closely correspond to the relevant visual content, and the target image coding features are obtained. The dynamic adjustment of the image coding weights is realized, and the reply text corresponding to the question text is generated according to the target image coding features, so that accurate answers can be generated for the questions.

[0017] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained through the structures particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a specific method flow chart of a picture-text question-answering method based on multimodal alignment provided in one embodiment of the present application; Figure 2 This is a flowchart of a specific method for inputting a scene image and a question text into a feature extraction module for feature extraction provided by an embodiment of the present application; Figure 3 This is a flowchart of a specific method for semantically aligning an image coding set and a dialogue coding set by using a weight matching module provided in an embodiment of the present application; Figure 4 This is a flowchart of a specific method for weighting and combining image feature items with text feature items in a dialogue coding set provided by an embodiment of the present application; Figure 5 This is a flowchart of a specific method for generating a reply text corresponding to a question text according to a target image coding feature provided by an embodiment of the present application; Figure 6 is a specific method flow chart of a picture-text question-answering method based on multimodal alignment provided by another embodiment of the present application; Figure 7 is a specific method flow chart of a picture-text question-answering method based on multimodal alignment provided by another embodiment of the present application; Figure 8 It is a flowchart of a specific method for obtaining a historical image feature set and a historical text feature set corresponding to a historical dialogue round in a preset multimodal feature alignment model provided by an embodiment of the present application; Fig. 9 is a specific method flow chart of a picture-text question-answering method based on multimodal alignment provided by another embodiment of the present application; Fig.10 is a system schematic diagram of a picture-text question-answering system based on multimodal alignment provided in one embodiment of the present application; Fig.11 It is a schematic diagram of a framework of a multimodal feature alignment model provided in an example of this application; Fig.12 This is a schematic diagram of the dynamic value of weights provided by an example of this application; Fig.13 It is a schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0019] This section will describe in detail the specific embodiments of the present invention. The preferred embodiments of the present invention are shown in the accompanying drawings. The purpose of the accompanying drawings is to supplement the description of the text part of the specification with graphics, so that people can intuitively and vividly understand each technical feature and the overall technical solution of the present invention, but it cannot be understood as a limitation on the scope of protection of the present invention.

[0020] In the description of the present invention, if there is a description of first and second, it is only for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0021] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0022] Visual question answering is a new task that has emerged in recent years with the rapid development of natural language processing and computer vision technology. It requires not only a deep understanding of the text, but also the accurate extraction of various attributes of the target from the image. The core of visual question answering lies in the effective interaction between visual features and text features. Currently, the more advanced models have mainly innovated in the fusion module. With the application of large language models (such as CLIP, ViT, DALL-E, etc.) in multi-round continuous dialogues, continuous question answering for visual content has also become feasible.

[0023] The current large language models can understand and generate text descriptions of image content, and perform in-depth reasoning on image content based on questions. However, in multimodal visual question answering, although the current large language models perform well when processing single-modal data, when processing complex interactive picture-text question answering tasks, they often suffer from performance degradation due to image quality issues and insufficient inter-modal fusion, which in turn affects the encoding effect of the language model, and ultimately affects the accuracy and contextual relevance of the question answering.

[0024] In order to solve the above problems, the present embodiment provides a method, system, electronic device and medium for image-text question and answer based on multimodal alignment, which has at least the following beneficial effects: in response to a dialogue instruction, the scene image and question text corresponding to the current dialogue round are obtained, and the initial weight of the scene image in the current dialogue round is determined, and then the historical image feature set and historical text feature set corresponding to the historical dialogue round of the preset multimodal feature alignment model are obtained, so as to facilitate the subsequent overall analysis of the historical images and historical questions in the dialogue, and then the scene image and question text are input into the feature extraction module for feature extraction, and the image features and text features are output to achieve accurate extraction of key information of the image and key information of the text, thereby improving the accuracy of analyzing the correlation between the problem and the image. The image features and initial weights are merged into the historical image feature set to obtain the image coding set, and the text features are merged into the historical text feature set to obtain the dialogue coding set, so that the model can understand and respond to contextual questions more accurately, ensuring that the logical coherence and contextual consistency of the information can be maintained when generating answers. The image coding set and the dialogue coding set are semantically aligned through the weight matching module, and the association weights between images and texts are dynamically adjusted to ensure that in continuous multi-round dialogues, each question of the user can closely correspond to the relevant visual content, obtain the target image coding features, realize dynamic adjustment of the image coding weights, and generate a reply text corresponding to the question text according to the target image coding features, so as to generate accurate answers to the questions.

[0025] Reference Figure 1 , Figure 1It is a specific method flow chart of a picture-text question-answering method based on multimodal alignment provided in one embodiment of the present application. The picture-text question-answering method based on multimodal alignment includes but is not limited to steps S101 to S106.

[0026] Step S101, in response to a dialogue instruction, obtaining a scene image and a question text corresponding to a current dialogue round, and determining an initial weight corresponding to the scene image.

[0027] In step S101 of some embodiments, in response to a dialogue instruction, the scene image and question text corresponding to the current dialogue round are obtained, that is, the scene image and question text involved in this round of dialogue are determined, and the initial weight corresponding to the scene image is determined to facilitate the subsequent feature fusion of the scene image and question text.

[0028] Step S102, obtaining a historical image feature set and a historical text feature set corresponding to a historical dialogue round of a preset multimodal feature alignment model, wherein the multimodal feature alignment model includes at least a feature extraction module and a weight matching module.

[0029] In step S102 of some embodiments, since in a scenario of multiple dialogue rounds, the questions asked by users in different dialogue rounds may be related to each other, and the relevant information of each dialogue round will be stored in the multimodal feature alignment model, the embodiment of the present application will obtain the historical image feature set and the historical text feature set corresponding to the historical dialogue rounds of the preset multimodal feature alignment model, so as to obtain the image features and text features of the historical dialogue rounds before the current dialogue round, so as to facilitate more accurate understanding and response to contextual questions in the future.

[0030] Step S103: input the scene image and question text into a feature extraction module for feature extraction, and output image features and text features.

[0031] In step S103 of some embodiments, the scene image and the question text are input into a feature extraction module for feature extraction, so as to capture the global information and the relationship between different image blocks of the original features, extract image features rich in global information, and capture the relationship between long-distance words, so as to better capture the semantic information in the text and output text features.

[0032] Step S104, merging the image features and the initial weights into the historical image feature set to obtain an image coding set, and merging the text features into the historical text feature set to obtain a dialogue coding set.

[0033] In step S104 of some embodiments, after obtaining the image features and text features, the embodiments of the present application merge the image features and the initial weights into a historical image feature set to obtain an image coding set, and merge the text features into a historical text feature set to obtain a dialogue coding set, thereby realizing real-time updating of feature items in the historical image feature set and the historical text feature set, so that subsequent modules can more accurately understand and respond to contextual questions.

[0034] Step S105, semantically aligning the image coding set and the dialogue coding set through a weight matching module to obtain target image coding features.

[0035] In step S105 of some embodiments, the image coding set and the dialogue coding set are semantically aligned through a weight matching module, and the dynamic semantic weight can adaptively assign different fusion weights to different input images and positions. By dynamically assigning weights, images and texts can be matched more accurately, and the dynamic weight assignment can take into account the relationship between images and texts in different contexts, so that the model can better understand and process complex scenes.

[0036] Step S106, generating a reply text corresponding to the question text according to the target image coding features.

[0037] In step S106 of some embodiments, a reply text corresponding to the question text is generated based on the target image coding features, thereby improving the accuracy of the question reply and generating a precise answer by effectively combining the image features with the text features.

[0038] In some embodiments, the embodiment of the present application will also verify the model performance of the multimodal feature alignment model through the visual language navigation dataset Room-to-Room (R2R), which is explained in detail below and will not be repeated in detail in the embodiment of the present application.

[0039] Reference Figure 2 , Figure 2 This is a flowchart of a specific method provided by an embodiment of the present application for inputting a scene image and a question text into a feature extraction module for feature extraction, and the method includes but is not limited to steps S201 to S203.

[0040] It should be noted that the feature extraction module includes an image encoder, a first transformer and a second transformer, wherein the first transformer interacts with the image encoder, and the second transformer serves as both a text encoder and a decoder. Specifically, the first transformer includes a self-attention module, a cross-attention module and a feedforward network, and the second transformer includes a self-attention module and a feedforward network.

[0041] Step S201: input a scene image into an image encoder so that the image encoder extracts features from the scene image and outputs original features.

[0042] In step S201 of some embodiments, the scene image is input into an image encoder so that the image encoder extracts features from the scene image. Specifically, the image encoder extracts features from the image through its convolutional layer, pooling layer and other structures, and further captures local and global information of the image, such as edges, textures, shapes, etc., to obtain original features, thereby being able to automatically extract key features of the image, remove noise, and retain key information that is meaningful to the task.

[0043] It should be noted that before the scene image is input into the image encoder, the embodiment of the present application may also perform preprocessing operations on the scene image. Specifically, the scene image is usually first subjected to standardization processing, such as resizing and normalizing the pixel values ​​to a certain range ([0,1] or [-1,1]), so that the image can adapt to the network input format, thereby avoiding input data anomalies during the model training process.

[0044] Step S202: Perform feature representation on the original features through a first transformer to obtain image features.

[0045] In step S202 of some embodiments, the original features are represented by a first transformer. Specifically, the first transformer uses a set of learnable query vectors to extract image features from a frozen image encoder to implement the interaction process between the first transformer and the image encoder, and then performs image segmentation on the extracted image features, cutting the image into small image blocks, and each image block is embedded into a vector of fixed dimension through a linear mapping as a feature representation of the image block. Since the Transformer architecture itself does not have the ability to process the order of the input sequence. Therefore, in order to allow the model to perceive the position of each image block, the embodiment of the present application will add the position code to the feature vector of each block, wherein the position code is usually generated by sine and cosine functions and added to the patch embedding to ensure that the Transformer can understand the relative position of each patch in the image. Next, the features of each image block are interacted with other image blocks through the self-attention module, and the attention weight is calculated, so that the model can pay attention to the connection between different parts of the image and capture the global spatial dependency. After the self-attention module, Transformer will further process and transform the features of each patch through a feedforward neural network to improve the nonlinear expression ability of the model and obtain image features, so as to capture the global information and the relationship between different image blocks of the original features and extract image features rich in global information.

[0046] It can be understood that the first transformer in the embodiment of the present application is Q-Former, which refers to a lightweight Transformer structure that interacts with the image encoder. The first transformer interacts with the image encoder through a set of learnable retrieval vectors, thereby being able to extract image features from the frozen image encoder.

[0047] Step S203, extracting features from the question text through the second transformer to obtain text features.

[0048] In step S203 of some embodiments, since the question text may contain a lot of information, in order to improve the answering efficiency, the embodiment of the present application will screen out the more critical information in the question text, and extract the features of the question text through the second transformer. Specifically, before the question text is input into the second transformer, the embodiment of the present application will also perform preprocessing operations on the question text, such as word segmentation, word embedding, position encoding, etc., to ensure that the second transformer can distinguish words in different positions. Then the question text is input into the second transformer, so that the second transformer converts the index of each word into a high-dimensional dense vector, wherein the embedding vector not only contains the semantic information of the vocabulary, but also adds the position information of the word in the sequence through the position encoding, and then calculates the relationship between words in the question text through the self-attention module of the second transformer, and dynamically adjusts the identification of each word. In order to enhance the representation ability of the model, the second transformer uses a multi-head attention mechanism, calculates multiple different attention heads in parallel, each head learns different types of relationships, and splices the outputs of these heads and obtains the final output through linear transformation. Afterwards, the output of the self-attention module is characterized by the feedforward network of the second transformer to obtain text features. The embodiment of the present application can easily capture the relationship between long-distance words through the second transformer and better capture the semantic information in the text.

[0049] Reference Figure 3 , Figure 3 This is a flowchart of a specific method for semantically aligning an image coding set and a dialogue coding set through a weight matching module provided in an embodiment of the present application. The method includes but is not limited to steps S301 to S302.

[0050] Step S301: for each image feature item in the image coding set, weighted combination is performed on the image feature item and the text feature item in the dialogue coding set to obtain an image weight corresponding to the image feature item.

[0051] In step S301 of some embodiments, in the process of semantically aligning the image coding set and the dialogue coding set through the weight matching module, for each image feature item in the image coding set, the image feature item and the text feature item in the dialogue coding set are weighted and combined. Through multiple rounds of weight matching, the model input pays more attention to key information and weakens the image input with low correlation, so as to obtain the image weight corresponding to the image feature item. The dynamic semantic weight can adaptively assign different fusion weights to different input images and positions. By dynamically assigning weights, images and texts can be matched more accurately, and the dynamic weight assignment can take into account the relationship between images and texts in different contexts, so that the model can better understand and process complex scenes.

[0052] Step S302: combining the image weight and the image feature corresponding to the image feature item to obtain the target image coding feature.

[0053] In step S302 of some embodiments, the image weights and image features corresponding to the image feature items are combined to achieve matching between the image weights and the image features, and obtain the target image coding features. Through the weight matching algorithm, it is ensured that in continuous multi-round conversations, the text can closely correspond to the relevant visual content, further enhancing the image-text fusion effect of the model.

[0054] It is worth noting that the image features of each dialogue input are , the image weight after semantic matching is Take the target image encoding feature as an example. . Weight matching extracts text features of key information from multiple rounds of conversations, and these text features are used to guide the reorganization of image encodings. The model dynamically adjusts the weights of image encodings based on the text features in the conversation, improving the image information that is most relevant to the current conversation content. This dynamic reorganization not only improves the consistency between the conversation and the image, but also greatly reduces processing time and computing resource consumption.

[0055] Reference Figure 4 , Figure 4 This is a flowchart of a specific method for weighting and combining image feature items with text feature items in a dialogue coding set provided by an embodiment of the present application. The method includes but is not limited to steps S401 to S403.

[0056] Step S401, calculating the similarity value between the image feature item and the text feature item through a preset similarity matrix.

[0057] Step S402: construct a similarity matrix according to the similarity values ​​of all image feature items.

[0058] Step S403: assigning semantic weights to the image feature items based on the similarity matrix to obtain image weights corresponding to the image feature items.

[0059] In steps S401 to S403 of some embodiments, in the process of weighted combination of image feature items and text feature items in the dialogue coding set, the embodiment of the present application first calculates the similarity values ​​of the image feature items and the text feature items through a preset similarity matrix. Specifically, the image feature items and the text feature items are feature normalized so that they have unit length, and then the similarity matrix M is used to calculate the dot product of the image features corresponding to the image feature items and the text features corresponding to the text feature items, so that the dot product of the normalized feature vectors can be directly used as a measure of cosine similarity. Thereafter, a similarity matrix is ​​constructed based on the similarity values ​​of all image feature items, wherein each element M[i][j] of M represents the similarity between the i-th image feature vector and the j-th text feature vector. The similarity calculation can combine different models The information of the two states (text and image) is fused together, so that the model can understand and process visual and language information at the same time, so that the image features can be effectively combined with the text features, which is convenient for the subsequent capture and understanding of the semantic connection between the image and the text. Finally, a weight is assigned to each image according to the calculated similarity. This weight reflects the degree of match between the image and the specific text features, thereby obtaining the dynamic semantic weight, that is, the image weight corresponding to the image feature item. The dynamic semantic weight can adaptively assign different fusion weights to different input images and positions. By dynamically assigning weights, images and texts can be matched more accurately, and the dynamic weight assignment can take into account the relationship between images and texts in different contexts, so that the model can better understand and process complex scenes, further enhancing the image-text fusion effect of the model.

[0060] Reference Figure 5 , Figure 5 This is a flowchart of a specific method for generating a reply text corresponding to a question text according to a target image coding feature provided by an embodiment of the present application. The method includes but is not limited to steps S501 to S502.

[0061] It should be noted that the multimodal feature alignment model also includes a large language model.

[0062] Step S501, feature representation is performed on the target image encoding features through the first transformer to obtain visual representation features.

[0063] Step S502: Input the visual representation features and text features into the large language model to predict the answer, and output the reply text corresponding to the question text.

[0064] In steps S501 to S502 of some embodiments, after obtaining the target image coding features, the embodiments of the present application pass the target image coding features after weight matching into the first transformer, so that the first transformer performs feature representation on the target image coding features to obtain visual representation features, thereby converting the target image coding features into a form that is more suitable for alignment with text features. Thereafter, the visual representation features and the text features are input into the large language model for answer prediction. The large language model generates an answer based on the visual representation features and the text features, and outputs a reply text corresponding to the question text, thereby improving the accuracy of the question response and obtaining a more accurate question and answer result.

[0065] It should be noted that the large language model in the embodiment of the present application can be a model based on an autoencoder, a recursive neural network model, a multimodal large language model, etc., and the embodiment of the present application does not make specific restrictions.

[0066] Reference Figure 6 , Figure 6 This is a specific method flow chart of a picture-text question-answering method based on multimodal alignment provided in another embodiment of the present application, and the method includes but is not limited to step S601.

[0067] It should be noted that the multimodal feature alignment model also includes a contrastive learning module, wherein the contrastive learning module is an image-text learning module. And step S601 occurs after the scene image and the question text are input into the feature extraction module for feature extraction, and the image features and the text features are output.

[0068] Step S601: aligning image features and text features through a contrast learning module.

[0069] In step S601 of some embodiments, feature alignment is performed on image features and text features through a contrastive learning module. Specifically, image features and text features are aligned through contrastive learning to maximize their mutual information, so that the image and text information are aligned in the same embedding space, thereby enabling effective cross-modal matching between image and text.

[0070] Reference Figure 7 , Figure 7 This is a specific method flow chart of a picture-text question-and-answer method based on multimodal alignment provided in another embodiment of the present application, and the method includes but is not limited to steps S701 to S703.

[0071] Step S701, designing an image loss function based on image features and preset training image features, and designing a text loss function based on text features and preset training text features.

[0072] Step S702: setting a sequential consistency loss function according to the image loss function and the text loss function.

[0073] Step S703: training a contrastive learning module according to a sequential consistency loss function and a preset modality alignment loss function.

[0074] In some embodiments, in step S701 to step S703, during the process of training and fine-tuning the multimodal feature alignment model, the present embodiment introduces two additional loss functions: the modality alignment loss function With sequential consistency loss function , sequential consistency loss function It consists of two sequential consistency losses for a single modality. Specifically, an image loss function is designed through image features and preset training image features, and a text loss function is designed through text features and preset training text features. Then, a sequential consistency loss function is set according to the image loss function and the text loss function. After that, a contrastive learning module is trained according to the sequential consistency loss function and the preset modality alignment loss function, wherein the modality alignment loss function is used to describe the feature alignment between the image and the text, thereby improving the accuracy of the feature alignment of the contrastive learning module, and the sequential consistency loss function helps the model to maintain the consistency of information when processing the joint representation of images and texts, and reduces the model's dependence on a specific order in the training data, thereby improving the stability of the model under different data orders. In order to further enhance the image-text fusion effect of the model, the embodiment of the present application introduces modality alignment loss and sequential consistency loss. These two loss functions help the model to better synchronize and align the features of image and text data during the training process, ensuring that the logical coherence and contextual consistency of the information can be maintained when generating answers.

[0075] Specifically, the modality alignment loss function is expressed as follows: ; in, and Respectively represent the image encoding features and text encoding features of the nth round of matching, Represents the temperature parameter. By introducing the modality alignment loss, the distinguishability of multi-round dialogue features is effectively enhanced, which can better guide modality alignment learning.

[0076] Text loss function It is expressed as follows: ; in, Used to characterize text features. It is used to characterize the features of the training text. By designing the text loss function, the text features are made as close as possible to the training text features. Similarly, the image loss function The definition process and text loss function The embodiments of the present application are not described in detail here.

[0077] Sequential consistency loss function It is expressed as follows: ; in, is the image loss function, is the text loss function.

[0078] The specific process of balancing the loss weights of the image loss function and the text loss function according to the sequential consistency loss function is as follows: ; in, and Used to balance the loss weights of each loss item, is the modality alignment loss function, is the sequential consistency loss function.

[0079] Reference Figure 8 , Figure 8 This is a flowchart of a specific method for obtaining a historical image feature set and a historical text feature set corresponding to a preset multimodal feature alignment model in a historical dialogue round, provided by an embodiment of the present application. The method includes but is not limited to step S801.

[0080] It should be noted that the multimodal feature alignment model includes an image frame memory module and a continuous dialogue memory module, wherein the image frame memory module is used to store image features in the context of multiple rounds of dialogues, and the continuous dialogue memory module is used to store text features in the context of multiple rounds of dialogues.

[0081] Step S801, obtaining a historical image feature set corresponding to a historical dialogue round through an image frame memory module, and obtaining a historical text feature set corresponding to a historical dialogue round through a continuous dialogue memory module.

[0082] In step S801 of some embodiments, a historical image feature set corresponding to historical dialogue rounds is obtained through an image frame memory module, and a historical text feature set corresponding to historical dialogue rounds is obtained through a continuous dialogue memory module, so as to facilitate subsequent overall analysis in combination with historical images and historical questions in the dialogue, thereby being able to better understand and utilize contextual information in multiple rounds of dialogue, thereby providing richer and more humane interactive services.

[0083] Reference Fig. 9 , Fig. 9 This is a specific method flow chart of a picture-text question-answering method based on multimodal alignment provided in another embodiment of the present application, and the method includes but is not limited to step S802.

[0084] It should be noted that step S802 occurs after the image features and initial weights are merged into the historical image feature set to obtain the image coding set, and the text features are merged into the historical text feature set to obtain the dialogue coding set.

[0085] Step S802, storing the image code set into the image frame memory module, and storing the dialogue code set into the continuous dialogue memory module.

[0086] In step S802 of some embodiments, after obtaining the image coding set and the dialogue coding set, the image coding set is stored in the image frame memory module, and the dialogue coding set is stored in the continuous dialogue memory module, so as to realize the storage of context information and record the scene images and question statements involved in multiple rounds of dialogue, so as to facilitate the subsequent model to analyze the problem in combination with the context and output accurate response results.

[0087] See also Fig.10 , Fig.10 It is a system schematic diagram of a picture-text question-answering system based on multimodal alignment provided by an embodiment of the present application.

[0088] In some embodiments, the present application also provides a picture-text question-answering system based on multimodal alignment, the system comprising: The data acquisition module 901 is used to obtain the scene image and question text corresponding to the current dialogue round in response to the dialogue instruction, and determine the initial weight corresponding to the scene image; A historical data acquisition module 902 is used to acquire a historical image feature set and a historical text feature set corresponding to a historical dialogue round of a preset multimodal feature alignment model, wherein the multimodal feature alignment model includes at least a feature extraction module and a weight matching module; A feature extraction module 903, used for inputting the scene image and the question text into the feature extraction module for feature extraction, and outputting image features and text features; A feature merging module 904 is used to merge the image features and the initial weights into the historical image feature set to obtain an image code set, and merge the text features into the historical text feature set to obtain a dialogue code set; A semantic alignment module 905 is used to semantically align the image coding set and the dialogue coding set through a weight matching module to obtain target image coding features; The reply generation module 906 is used to generate a reply text corresponding to the question text according to the target image coding features.

[0089] In some embodiments, the image-text question-answering system based on multimodal alignment also includes a model training module and a feature alignment module, wherein the model training module is used to design an image loss function through image features and preset training image features, and to design a text loss function through text features and preset training text features; to set a sequential consistency loss function according to the image loss function and the text loss function; and to train a contrastive learning module according to the sequential consistency loss function and the preset modality alignment loss function. The feature alignment module is used to perform feature alignment of image features and text features through the contrastive learning module.

[0090] The specific implementation of the multimodal alignment-based picture-text question-answering system is basically the same as the specific implementation of the multimodal alignment-based picture-text question-answering method described above, and will not be repeated here.

[0091] In order to more clearly illustrate the above-mentioned image-text question-answering method based on multimodal alignment, a specific example is given below.

[0092] Example 1: See also Fig.11 , Fig.11 This is a schematic diagram of the framework of a multimodal feature alignment model provided as an example in this application.

[0093] In some embodiments, the multimodal feature alignment model includes modules such as an image encoder, a first transformer, a second transformer, a large language model, a contrastive learning module, an image-text matching module, a weight matching module, an image frame memory module, and a continuous dialogue memory module.

[0094] The input part of the model uses a Querying Transformer (Q-Former) similar to BLIP2 (Bootstrapping Language-Image Pre-training2), which is the first transformer of the embodiment of the present application. The first transformer uses a set of learnable query vectors to extract image features from a frozen image encoder. The first transformer on the left interacts with the image encoder to extract visual representations, and the second transformer on the right serves as both a text encoder and decoder. The two transformers share the same self-attention module, and the interaction between the image itself and the text itself is realized through the self-attention mechanism, and the interaction between the image text query and the result is realized through cross attention.

[0095] In a single question answering, the contrastive learning module learns to align image features and text features to maximize the information of both. For the input image, the features are first extracted through the image encoder, and then passed to the Q-former (the first transformer) to convert it into a feature representation, which is then used in the text. The visual representation inserts labels into the image feature representation.

[0096] The memory module in multiple rounds of dialogue is recorded in the form of labels. In each round of dialogue, the user is the user, and the model framework reply is the assistant. The continuous dialogue memory is: QA1:user: Which room is this? How do I get to the living room? Visual Representation1.

[0097] Assistant: This is me. Turn around and go out the door behind you.

[0098] QA2:user:Where is the sofa? Are there any obstacles? Visual representation2.

[0099] assistant: The sofa is in front and there is a coffee table on the right side of the path.

[0100] The image memory is formed by merging the features extracted by the image encoder, i.e., [Img_1, Img_2, …Img_i].

[0101] It is understandable that the embodiment of the present application saves context information through an image frame memory module and a continuous dialogue memory module between multiple rounds of dialogues, and uses an image-text weight matching module to introduce a lightweight image-text matching module. This image-text matching module is used to evaluate and optimize the matching degree between the dialogue content and the image encoding. The model can more accurately determine which image encodings are most relevant to the current dialogue content, thereby performing more effective image and text reorganization. By dynamically determining the weights between multiple rounds of dialogues, the emphasis information during multiple rounds of questions and answers is reflected, and more accurate multiple rounds of question and answer results are obtained. The dynamic determination of weights between multiple rounds of dialogues is specifically described below.

[0102] See also Fig.12 , Fig.12 This is a schematic diagram of the dynamic value of weights provided by an example of this application.

[0103] In multi-round question answering, the multimodal feature alignment model includes a dynamic weight matching module, namely the weight matching module. The matching of dialogue and image encoding is as follows: Fig.12 As shown, the picture encoding features of each dialogue input are , the text features are , the initial weight corresponding to the image is , the image encoding feature is initially , the text features are matched by similarity matrix M Then get the dynamic semantic weight , that is, the image weight after semantic matching is obtained, and then the image encoding feature after matching is obtained as , and then the weight-matched image encoding is passed to the first transformer to be converted into feature representation. Through multiple rounds of weight matching, the model input focuses more on key information and weakens the image input with low relevance.

[0104] It should be noted that weight matching extracts the encoding information of key information from multiple rounds of dialogue, and this text information is used to guide the reorganization of image encoding. The model dynamically adjusts the weight of image encoding based on the key information in the dialogue, and improves the image information most relevant to the current dialogue content. This dynamic reorganization not only improves the consistency between dialogue and image, but also greatly reduces processing time and computing resource consumption.

[0105] In the training and fine-tuning process, the embodiment of the present application also introduces two additional loss functions, modality alignment loss and sequential consistency loss. In modality alignment loss, the infoNCE loss function is used to describe the feature alignment between image and text: ; in, and Respectively represent the image encoding features and text encoding features of the nth round of matching, Represents the temperature parameter. By introducing the modality alignment loss, the distinguishability of multi-round dialogue features is effectively enhanced, which can better guide modality alignment learning.

[0106] At the same time, a sequential consistency loss is introduced, which consists of two sequential consistency losses for a single modality. Taking the text modality as an example, in the nth round of dialogue, the design loss , make the text prompt feature and The training features are as close as possible, that is: ; Similarly, define the image loss function , based on the loss function and , we get the sequential consistency loss function : ; in, is the image loss function, is the text loss function.

[0107] The specific process of balancing the loss weights of the image loss function and the text loss function according to the sequential consistency loss function is as follows: ; in, and Used to balance the loss weights of each loss item, is the modality alignment loss function, is the sequential consistency loss function.

[0108] In addition, the embodiment of the present application will also conduct model evaluation through the visual language navigation dataset Room-to-Room (R2R). The R2R dataset is a navigation dataset containing English instructions, with 10,800 panoramic views, 7,189 trajectories and 21,567 navigation multi-round question-and-answer instructions.

[0109] The R2R dataset is divided into a training set Train, a validation set Val Seen for seen scenes, a validation set Val Unseen for unseen scenes, and a test set Test Unseen for unseen scenes. Train, Val Seen, Val Unseen, and Test Unseen contain 61, 56, 11, and 18 house scenes, respectively. For the R2R dataset, the following common evaluation indicators are used: Trajectory Length (TL) calculates the average length of the trajectory.

[0110] Navigation Error (NE) calculates the distance between the target point and the stopping position.

[0111] Success Rate (SR) is the proportion of navigation trajectories that reach the end point.

[0112] The success rate weighted by Path Length (SPL) is a compromise between SR and TL.

[0113] Two encoders are used to obtain visual features for testing, one is the ResNet-152 model, and the other is the visual encoder in the CLIP model. Considering the differences in instruction lengths in different datasets, on the R2R dataset, the action prompt set for each instruction is set to contain a maximum of 60 action prompts.

[0114] Table 1 Experimental results on R2R

[0115] The above table shows the comparison between this method and existing methods on the R2R dataset. It can be observed from the table that the method based on ResNet-152 visual features achieves the best performance in all comparison items, which shows the effectiveness of the method. In addition, by introducing CLIP visual features, the performance of both models on Val Unseen is improved, but the performance on Val Seen and Test Unseen is reduced. However, when using CLIP, the method in this paper exceeds CLIP in all evaluation indicators, which shows the effectiveness of the method.

[0116] In some embodiments, the embodiments of the present application integrate advanced image processing technology and natural language processing capabilities, and accurately parse image content and text information through deep learning models. The framework design aims to optimize the process of extracting visual features from images, and effectively combine these features with text data to generate accurate answers. At the same time, a multimodal semantic alignment module for multi-round question-and-answer image and text weight matching is introduced. Through the weight matching algorithm, it is ensured that in continuous multi-round conversations, the text can closely correspond to the relevant visual content. By dynamically adjusting the association weights between images and texts, contextual questions can be understood and responded to more accurately. In addition, modal alignment loss and sequential consistency loss are introduced to further enhance the image-text fusion effect of the model, ensuring that the logical coherence and contextual consistency of the information can be maintained when generating answers.

[0117] Reference Fig.13 , taking the control processor 1001 and memory 1002 in the electronic device 1000 as an example that can be connected via a bus. The memory 1002, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory 1002 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk memory, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1002 may optionally include a memory remotely arranged relative to the control processor 1001, and these remote memories may be connected to the electronic device 1000 via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0118] Those skilled in the art will understand that Fig.13 The device structure shown in the figure does not constitute a limitation on the electronic device 1000, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0119] The embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are executed by one or more control processors, for example, Fig.13One of the control processors 1001 executes, which can enable the above-mentioned one or more control processors to execute the image-text question-answering method based on multimodal alignment in the above-mentioned method embodiment.

[0120] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0121] It will be appreciated by those skilled in the art that all or some of the steps and systems in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

Claims

1. A picture-text question answering method based on multimodal alignment, characterized in that: include: In response to the dialogue instruction, a scene image and a question text corresponding to the current dialogue round are obtained, and an initial weight corresponding to the scene image is determined; Obtaining a historical image feature set and a historical text feature set corresponding to a historical dialogue round of a preset multimodal feature alignment model, wherein the multimodal feature alignment model at least includes a feature extraction module and a weight matching module; Inputting the scene image and the question text into the feature extraction module for feature extraction, and outputting image features and text features; Merging the image features and the initial weights into the historical image feature set to obtain an image encoding set, and merging the text features into the historical text feature set to obtain a dialogue encoding set; Semantically aligning the image coding set and the dialogue coding set by the weight matching module to obtain target image coding features; Generate a reply text corresponding to the question text according to the target image encoding feature.

2. The method for image-text question answering based on multimodal alignment according to claim 1, characterized in that: The feature extraction module includes an image encoder, a first transformer and a second transformer; the scene image and the question text are input into the feature extraction module for feature extraction, and the image features and text features are output, including: Inputting the scene image into the image encoder so that the image encoder extracts features from the scene image and outputs original features; Performing feature representation on the original feature by the first transformer to obtain image features; The second transformer is used to extract features from the question text to obtain text features.

3. The method for image-text question answering based on multimodal alignment according to claim 1, characterized in that: The semantically aligning the image coding set and the dialogue coding set by the weight matching module to obtain target image coding features includes: For each image feature item in the image coding set, weighted combination is performed on the image feature item and the text feature item in the dialogue coding set to obtain an image weight corresponding to the image feature item; The image weight is combined with the image feature corresponding to the image feature item to obtain a target image coding feature.

4. The method for image-text question answering based on multimodal alignment according to claim 3, characterized in that: The step of weighting the image feature item and the text feature item in the dialogue coding set to obtain an image weight corresponding to the image feature item includes: Calculating the similarity value between the image feature item and the text feature item through a preset similarity matrix; Constructing a similarity matrix according to the similarity values ​​of all the image feature items; Semantic weights are assigned to the image feature items based on the similarity matrix to obtain image weights corresponding to the image feature items.

5. The method for image-text question answering based on multimodal alignment according to claim 2, characterized in that: The multimodal feature alignment model also includes a large language model; the step of generating a reply text corresponding to the question text according to the target image encoding feature includes: Performing feature representation on the target image encoding feature by the first transformer to obtain a visual representation feature; The visual representation features and the text features are input into the large language model to predict the answer, and a reply text corresponding to the question text is output.

6. The method for image-text question answering based on multimodal alignment according to claim 1, characterized in that: The multimodal feature alignment model also includes a contrast learning module; after the scene image and the question text are input into the feature extraction module for feature extraction and the image features and the text features are output, the method further includes: The image features and the text features are aligned through the contrastive learning module.

7. The method for image-text question answering based on multimodal alignment according to claim 6, characterized in that: The method further comprises: An image loss function is designed by using the image features and the preset training image features, and a text loss function is designed by using the text features and the preset training text features; Setting a sequential consistency loss function according to the image loss function and the text loss function; The contrastive learning module is trained according to the sequential consistency loss function and a preset modality alignment loss function.

8. The method for image-text question answering based on multimodal alignment according to claim 1, characterized in that: The multimodal feature alignment model includes an image frame memory module and a continuous dialogue memory module; The step of obtaining a historical image feature set and a historical text feature set corresponding to a historical dialogue turn of a preset multimodal feature alignment model includes: Acquire a historical image feature set corresponding to a historical dialogue round through the image frame memory module, and acquire a historical text feature set corresponding to a historical dialogue round through the continuous dialogue memory module; After merging the image features and the initial weights into the historical image feature set to obtain an image encoding set, and merging the text features into the historical text feature set to obtain a dialogue encoding set, the method further includes: The image code set is stored in the image frame memory module, and the dialogue code set is stored in the continuous dialogue memory module.

9. A picture-text question answering system based on multimodal alignment, characterized in that: include: A data acquisition module, for acquiring a scene image and a question text corresponding to a current dialogue round in response to a dialogue instruction, and determining an initial weight corresponding to the scene image; A historical data acquisition module, used to acquire a historical image feature set and a historical text feature set corresponding to a historical dialogue round of a preset multimodal feature alignment model, wherein the multimodal feature alignment model includes at least a feature extraction module and a weight matching module; A feature extraction module, used for inputting the scene image and the question text into the feature extraction module for feature extraction, and outputting image features and text features; A feature merging module, used for merging the image features and the initial weights into the historical image feature set to obtain an image coding set, and merging the text features into the historical text feature set to obtain a dialogue coding set; A semantic alignment module, used for performing semantic alignment on the image coding set and the dialogue coding set through the weight matching module to obtain target image coding features; A reply generation module is used to generate a reply text corresponding to the question text according to the target image coding features.

10. An electronic device, characterized in that: It includes at least one processor and a memory for communicating with the at least one processor; the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image-text question-answering method based on multimodal alignment as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the image-text question-answering method based on multimodal alignment as described in any one of claims 1 to 8.

Images