Knowledge fusion multi-modal interaction method and apparatus based on improved alignment method

Abstract

A knowledge fusion multi-modal interaction method and apparatus based on an improved alignment method. The method comprises: acquiring multi-modal data (S101); constructing a query transformation model (S102); on the basis of a multi-stream feature extraction method, performing improved processing on a vision transformer to obtain a multi-scale vision transformer (S103); on the basis of a joint text method, training the multi-scale vision transformer to obtain a target vision transformer (S104); freezing the target vision transformer, and on the basis of the target vision transformer, performing vision-text alignment training processing on the query transformation model to obtain a target query transformation model (S105); connecting the target query transformation model to a language model to obtain a knowledge fusion multi-modal interaction model (S106); and inputting the multi-modal data into the knowledge fusion multi-modal interaction model to obtain an interaction result (S107). In the embodiments of the present invention, the knowledge fusion multi-modal interaction model aligns visual representation with text representation, so that the mutual information between the visual representation and the text representation can be maximized, and the present invention can be widely applied to the technical field of artificial intelligence.

Description

A knowledge fusion multimodal interaction method and device based on improved alignment method Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a knowledge fusion multimodal interaction method and device based on an improved alignment method. Background Art

[0002] In recent years, the fields of computer vision and natural language processing have both achieved rapid progress. However, many practical problems are inherently multimodal, involving several different forms of data, such as images and text. Most human-computer interaction methods in related technologies rely solely on text or voice interaction, and are unable to process multimodal data such as images and text, resulting in poor interaction efficiency. In summary, the technical issues existing in related technologies urgently need to be addressed.

[0003] Summary of the Invention

[0004] In view of this, an embodiment of the present invention provides a knowledge fusion multimodal interaction method and device based on an improved alignment method to improve the efficiency of human-computer interaction.

[0005] In one aspect, the present invention provides a knowledge fusion multimodal interaction method based on an improved alignment method, the multimodal interaction method comprising:

[0006] Acquire multimodal data;

[0007] Constructing a query transformation model, wherein the query transformation model includes a visual encoder, a visual transformer, and a text transformer;

[0008] The visual transformer is improved according to a multi-stream feature extraction method to obtain a multi-scale visual transformer;

[0009] Training the multi-scale visual transformer according to the joint text method to obtain a target visual transformer;

[0010] Freezing the target visual transformer, and performing visual-text alignment training on the query transformation model according to the target visual transformer to obtain a target query transformation model;

[0011] Connecting the target query transformation model to the language model to obtain a knowledge fusion multimodal interaction model;

[0012] The multimodal data is input into the knowledge fusion multimodal interaction model to obtain an interaction result.

[0013] Optionally, constructing the query transformation model includes:

[0014] Freezing the visual encoder and connecting the frozen visual encoder to the visual converter;

[0015] The visual transformer and the text transformer interact with each other through a shared self-attention layer to construct a query transformation model.

[0016] Optionally, the visual transformer is improved according to the multi-stream feature extraction method to obtain a multi-scale visual transformer, including:

[0017] Obtain multiple video streams through the visual encoder output;

[0018] A residual network is added to the visual transformer to obtain a multi-scale visual transformer. The residual network is used to perform multi-stream feature extraction processing on the multiple video streams to obtain multi-scale features. The residual network includes a residual block, a spatial pyramid pooling layer and a fully connected layer.

[0019] Optionally, performing multi-stream feature extraction processing on the multiple video streams to obtain multi-scale features includes:

[0020] Performing convolution and identity mapping processing on the multiple video streams respectively through the residual block to obtain a mapping vector set;

[0021] Performing local information extraction processing on the mapping vector set at different scales through the spatial pyramid pooling layer to obtain a video stream feature vector set;

[0022] Performing cascade processing on the video stream feature vector set to obtain a cascade data set;

[0023] The cascade data set is mapped through the fully connected layer to obtain multi-scale features.

[0024] Optionally, the training process of the multi-scale visual transformer according to the joint text method to obtain a target visual transformer includes:

[0025] Acquire training data, where the training data includes the multi-scale features and text features;

[0026] Performing cascade fusion processing on the training data to obtain an input feature matrix;

[0027] Performing multi-head self-attention processing on the input feature matrix through the self-attention layer to obtain an output feature matrix;

[0028] Performing feature extraction processing on the output feature matrix through a feedforward neural network to obtain splicing features;

[0029] Performing multimodal fusion processing on the splicing features in combination with the word segmentation model to obtain fusion features;

[0030] The fusion features are updated to training data, and the process returns to the step of performing cascade fusion processing on the training data to obtain an input feature matrix, until the number of cycles reaches a preset threshold, thereby obtaining a target visual transformer.

[0031] Optionally, the combined word segmentation model performs multimodal fusion processing on the splicing features to obtain fusion features, including:

[0032] Performing word segmentation processing on the training data using the word segmentation model to obtain word segmentation features;

[0033] Projecting the word segmentation features through a convolutional neural network to obtain feature labels;

[0034] Performing relationship modeling on the feature tags through an autonomous attention mechanism to obtain feature representation;

[0035] The feature representation and the splicing feature are spliced ​​together to obtain a fusion feature.

[0036] Optionally, performing visual-text alignment training processing on the query transformation model according to the target visual transformer to obtain the target query transformation model includes:

[0037] obtaining sample data, the sample data comprising a visual representation and a textual representation;

[0038] Extracting and processing the sample data by the target visual transformer to obtain a query representation;

[0039] Aligning the query representation with the text representation to obtain text similarity;

[0040] The parameters of the text transformer are updated according to the text similarity to obtain a target query transformation model.

[0041] On the other hand, an embodiment of the present invention further provides a multimodal interaction device based on an improved alignment method, the device comprising:

[0042] The first module is used to obtain multimodal data;

[0043] The second module is used to build a query transformation model, which includes a visual encoder, a visual transformer and a text transformer;

[0044] The third module is used to improve the visual transformer according to the multi-stream feature extraction method to obtain a multi-scale visual transformer;

[0045] A fourth module is configured to train the multi-scale visual transformer according to a joint text method to obtain a target visual transformer;

[0046] a fifth module, configured to freeze the target visual transformer and perform visual-text alignment training on the query transformation model according to the target visual transformer to obtain a target query transformation model;

[0047] A sixth module is configured to connect the target query transformation model to a language model to obtain a knowledge fusion multimodal interaction model;

[0048] The seventh module is used to input the multimodal data into the knowledge fusion multimodal interaction model to obtain an interaction result.

[0049] On the other hand, an embodiment of the present invention further discloses an electronic device, including a processor and a memory;

[0050] The memory is used to store programs;

[0051] The processor executes the program to implement the method described above.

[0052] On the other hand, an embodiment of the present invention further discloses a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0053] In another aspect, embodiments of the present invention further disclose a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0054] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects: the embodiment of the present invention improves the visual transformer according to the multi-stream feature extraction method to obtain a multi-scale visual transformer, which can perform different types of feature extraction on video frames; in addition, the embodiment of the present invention performs visual text alignment training on the query transformation model according to the target visual transformer, which can improve the multimodal alignment effect on the video; furthermore, the embodiment of the present invention obtains a knowledge fusion multimodal interaction model by connecting the target query transformation model to the language model, which can improve the multimodal interaction capability on the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0056] FIG1 is a flow chart of a knowledge fusion multimodal interaction method based on an improved alignment method provided in an embodiment of the present application;

[0057] FIG2 is a schematic diagram of the structure of a query transformation model provided in an embodiment of the present application;

[0058] FIG3 is a schematic diagram of the structure of a multi-scale visual converter provided in an embodiment of the present application;

[0059] FIG4 is a schematic diagram of a training method of a multi-scale visual transformer provided in an embodiment of the present application;

[0060] FIG5 is a schematic diagram of the structure of a knowledge fusion multimodal interaction device based on an improved alignment method provided in an embodiment of the present application;

[0061] FIG6 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0063] The embodiments of the present application provide a knowledge fusion multimodal interaction method. The interaction method in the embodiments of the present application can be applied to a terminal or a server, or can be software running on a terminal or a server. The terminal can be a tablet computer, a laptop computer, a desktop computer, etc., but is not limited to such. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0064] 1 , an embodiment of the present invention provides a knowledge fusion multimodal interaction method based on an improved alignment method, the multimodal interaction method comprising:

[0065] S101, acquiring multimodal data;

[0066] S102, constructing a query transformation model, wherein the query transformation model includes a visual encoder, a visual transformer, and a text transformer;

[0067] S103, improving the visual transformer according to the multi-stream feature extraction method to obtain a multi-scale visual transformer;

[0068] S104, training the multi-scale visual transformer according to the joint text method to obtain a target visual transformer;

[0069] S105, freezing the target visual transformer, and performing visual-text alignment training on the query transformation model according to the target visual transformer to obtain a target query transformation model;

[0070] S106, connecting the target query transformation model to the language model to obtain a knowledge fusion multimodal interaction model;

[0071] S107: Input the multimodal data into the knowledge fusion multimodal interaction model to obtain an interaction result.

[0072] In an embodiment of the present invention, the multimodal interaction method can be applied to a terminal for human-computer interaction. For example, multimodal data is input through a target object on a computer, and the terminal performs corresponding processing through the multimodal interaction method of the embodiment of the present invention, outputting corresponding interaction results. The interaction results can be corresponding answers, suggestions, etc. The embodiment of the present invention first obtains multimodal data, which includes images, text, videos, and other data, which are uploaded or input by the target object. The embodiment of the present invention mainly analyzes and processes the multimodal data uploaded by the target object through a knowledge fusion multimodal interaction model, specifically by constructing a query transformation model, which includes a visual encoder, a visual transformer, and a text transformer. In order to process multi-stream data, the embodiment of the present invention improves the visual transformer through a multi-stream feature extraction method to obtain a multi-scale visual transformer. The multi-scale visual transformer can simultaneously consider multiple different scales of the video input in time and space. The multi-scale visual transformer is then trained according to the joint text method to obtain a target visual transformer. The target visual transformer is then frozen, and the query transformation model is trained for visual-text alignment based on the target visual transformer to obtain a target query transformation model. In the visual-text alignment learning task, the visual representation and the text representation need to be aligned to maximize the mutual information between them. The target visual transformer is fixed and no longer trained. Compared with training the visual transformer and the text transformer at the same time in an end-to-end model, this method can reduce model parameters and computational complexity, and can make better use of the trained visual transformer. Finally, the target query transformation model is connected to the language model to obtain a knowledge fusion multimodal interaction model. By inputting the multimodal data into the knowledge fusion multimodal interaction model, an interaction result is obtained. For example, the target object uploads a video to the terminal and inputs the corresponding text content. The video and text content are analyzed by the knowledge fusion multimodal interaction model in an embodiment of the present invention, and the corresponding interaction result is output.

[0073] It should be noted that in various specific implementations of the present application, when it comes to the need to perform relevant processing based on the target object's information, the target object's behavioral data, the target object's historical data, and the target object's location information, etc., which are related to the target object's identity or characteristics, the target object's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain sensitive information of the target object, the target object's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the target object's separate permission or consent, the necessary target object-related data for the normal operation of the embodiment of the present application will be obtained.

[0074] As an optional implementation, the constructing of the query transformation model includes:

[0075] Freezing the visual encoder and connecting the frozen visual encoder to the visual converter;

[0076] The visual transformer and the text transformer interact with each other through a shared self-attention layer to construct a query transformation model.

[0077] In an embodiment of the present invention, referring to FIG2 , the query transformation model includes a visual encoder and a Q-Former model. The Q-Former model is a trainable module that bridges the gap between a frozen visual encoder and a fixed language model (LLM). It can extract a fixed number of output features from the visual encoder, independent of the input image resolution. The Q-Former consists of two transformer submodules, a visual transformer and a text transformer, which share the same self-attention layer. One of the transformers is an image transformer that interacts with the frozen visual encoder and is used for visual feature extraction. The visual transformer comprises a self-attention layer, a cross-attention layer, and a feedforward layer. The visual encoder is frozen and then connected to the visual transformer. The other transformer is a text transformer that can serve as both a text encoder and a text decoder and comprises a self-attention layer and a feedforward layer. The visual transformer and the text transformer interact via a shared self-attention layer to construct a query transformation model. This embodiment of the present invention creates a set of learnable embeddings as input query embeddings for the visual transformer. Queries interact with each other via self-attention layers and with frozen image features via cross-attention layers (interposed within each transformer block). Depending on the pre-training task, embodiments of the present invention use different self-attention masks to control query-text interaction. This bottleneck architecture, combined with the pre-training objective of embodiments of the present invention, forces queries to extract the most relevant visual information from the text.

[0078] As a further optional implementation, the visual transformer is improved according to the multi-stream feature extraction method to obtain a multi-scale visual transformer, including:

[0079] Obtain multiple video streams through the visual encoder output;

[0080] A residual network is added to the visual transformer to obtain a multi-scale visual transformer. The residual network is used to perform multi-stream feature extraction processing on the multiple video streams to obtain multi-scale features. The residual network includes a residual block, a spatial pyramid pooling layer and a fully connected layer.

[0081] In an embodiment of the present invention, the visual transformer is improved. The construction process of this visual transformer is based on the idea of ​​multi-scale feature extraction, which can simultaneously consider multiple different scales of video input in time and space. Specifically, the embodiment of the present invention uses a multi-stream method, each stream extracts features for different spatial and temporal scales, and then these features are cascaded and processed together with the text / question input. The multi-scale visual transformer is mainly obtained by adding a residual network to the visual transformer. The residual network is used to perform multi-stream feature extraction processing on the multiple video streams to obtain multi-scale features. The residual network includes a residual block, a spatial pyramid pooling layer and a fully connected layer. Specifically, the embodiment of the present invention uses the X3D network as the basic model, which has performed well in many video intelligent answering technology (VideoQA) tasks. Referring to Figure 3, the embodiment of the present invention uses a residual network (ResNet) to extract features from each video stream, and uses three video streams of different scales to obtain feature representations of multiple visual modalities.

[0082] Further as an optional implementation manner, performing multi-stream feature extraction processing on the multiple video streams to obtain multi-scale features includes:

[0083] Performing convolution and identity mapping processing on the multiple video streams respectively through the residual block to obtain a mapping vector set;

[0084] Performing local information extraction processing on the mapping vector set at different scales through the spatial pyramid pooling layer to obtain a video stream feature vector set;

[0085] Performing cascade processing on the video stream feature vector set to obtain a cascade data set;

[0086] The cascade data set is mapped through the fully connected layer to obtain multi-scale features.

[0087] In this embodiment of the present invention, a set of residual blocks is used, where each block includes a 3×3 convolution kernel and an identity mapping layer. The residual blocks are used to perform convolution and identity mapping on the multiple video streams to obtain a set of mapping vectors. This process can be expressed as follows: vi =ResNet(x vi )

[0088] where x viRepresents the input of the i-th video stream, that is, it corresponds to multiple frames of video in the stream. After each residual block, a spatial pyramid pooling layer is added. The spatial pyramid pooling layer extracts local information from the mapping vector set at different scales to obtain a set of video stream feature vectors to capture local information at different scales. Finally, the features generated by each stream are cascaded, and the video stream feature vector set is cascaded to obtain a cascade dataset, and a fully connected layer is used to map them into a fixed-length vector f v , that is: f v =FCN(concat(f v1 ,f v2 ,f v3 ))

[0089] Here, concat means concatenating three feature sets together, and FCN means the fully connected layer used. The concatenated dataset is mapped through the fully connected layer to obtain multi-scale features, where f v It is a multi-scale feature for the entire video clip.

[0090] As a further optional implementation, the training process of the multi-scale visual transformer according to the joint text method to obtain a target visual transformer includes:

[0091] Acquire training data, where the training data includes the multi-scale features and text features;

[0092] Performing cascade fusion processing on the training data to obtain an input feature matrix;

[0093] Performing multi-head self-attention processing on the input feature matrix through the self-attention layer to obtain an output feature matrix;

[0094] Performing feature extraction processing on the output feature matrix through a feedforward neural network to obtain splicing features;

[0095] Performing multimodal fusion processing on the splicing features in combination with the word segmentation model to obtain fusion features;

[0096] The fusion features are updated to training data, and the process returns to the step of performing cascade fusion processing on the training data to obtain an input feature matrix, until the number of cycles reaches a preset threshold, thereby obtaining a target visual transformer.

[0097] In an embodiment of the present invention, referring to FIG4 , the multi-scale visual transformer is trained and further optimized using the combined text method. First, training data is obtained, the training data including the multi-scale features and text features. Then, the training data is cascaded and fused to obtain an input feature matrix. Specifically, the feature f generated by the video encoder is converted into v and embedding of text / question features f t Through cascade operation, we can get an input feature matrix X0 with a dimension of (M+N)×D:

[0098] Where M represents the number of video features, N represents the number of text features, and D represents the dimension of the features.

[0099] Next, the embodiment of the present invention uses a multi-layer Transformer model to further process the feature matrix X0. In the i-th layer Transformer, the input feature matrix X i 1. Apply a self-attention mechanism to capture the relationship between different features. The input feature matrix is ​​processed by the self-attention layer to obtain the output feature matrix Z. i , which can be expressed as: Z i =MultiHead(X i 1)

[0100] MultiHead represents the multi-head attention mechanism, which divides the input matrix into several sub-matrices and applies an attention mechanism to each sub-matrix to obtain multiple feature maps. The input matrix is ​​divided into K sub-matrices, then MultiHead can be expressed as: MultiHead(X)=concat(head1,…,head K )

[0101] Among them head k Represents the feature map obtained after applying the attention mechanism to the kth sub-matrix. Next, the embodiment of the present invention performs feature extraction processing on the output feature matrix through a forward neural network to obtain splicing features, specifically by using a forward neural network (Feedforward Network) to extract the output feature matrix Z i Further processing is performed to extract more useful information. The output of this feedforward neural network is H i , it can be expressed as: H i =FFN(Z i )

[0102] FFN represents a feedforward neural network, which consists of two fully connected layers, each of which contains an activation function and some regularization operations. Finally, the embodiment of the present invention combines the word segmentation model to perform multimodal fusion processing on the spliced ​​features to obtain fused features. In each Transformer layer, the word segmentation model (TokenLearner) is used to generate informative tokens. In the i-th layer Transformer, the input feature matrix X is first converted to i 1 Pass it to the TokenLearner of the video part and the text / question part respectively, so as to obtain a set of informative tokens respectively and Then, these tokens are concatenated and further processed through a fully connected layer to obtain an informative token T with a dimension of D. i :

[0103] FCN2 represents a network with two fully connected layers, which is used to map the concatenated tokens to a fixed-length vector. Finally, in each Transformer layer, the informative token T i And the feature matrix H output by the previous layer Transformer i Splicing is performed to obtain fusion features. The splicing operation can be expressed as: X i =concat(H i ,T i )

[0104] In this embodiment of the present invention, the fused features are updated as training data and input into the next Transformer layer for processing. This returns to the step of cascading the training data to obtain the input feature matrix, and this process continues until the number of cycles reaches a preset threshold, resulting in the target visual transformer. Typically, the number of fine-tuning cycles, i.e., the preset threshold, can range from several hundred to several thousand.

[0105] As an optional implementation, the word segmentation model is combined to perform multimodal fusion processing on the splicing features to obtain fusion features, including:

[0106] Performing word segmentation processing on the training data using the word segmentation model to obtain word segmentation features;

[0107] Projecting the word segmentation features through a convolutional neural network to obtain feature labels;

[0108] Performing relationship modeling on the feature tags through an autonomous attention mechanism to obtain feature representation;

[0109] The feature representation and the splicing feature are spliced ​​together to obtain a fusion feature.

[0110] In an embodiment of the present invention, tokenization is learned to compress the feature representation of each modality and convert it into a set of meaningful token representations, enabling better multimodal fusion. Specifically, this method employs a Token Learner model (segmentation module) to learn the tokens of each modality and convert them into token representations of the same dimension. This significantly reduces the number of feature representations, thereby improving the efficiency and accuracy of the model. Furthermore, because Token Learner is an adaptive mechanism, it can automatically select the most important information based on the input data, further enhancing the model's effectiveness. For video input, multi-scale, multi-stream features f{vi} are used, where i is the index of the feature stream. First, the training data is segmented using the segmentation model to obtain segmentation features. Specifically, the text features of the segmented words are summed with the video features to obtain a composite feature f = f_{vi} + r. This feature is then projected onto an N-dimensional token representation space using a convolutional neural network (CNN). A self-attention mechanism is used to model the relationships between different tokens to obtain a feature representation. This feature representation is then concatenated with the concatenated features to obtain a fused feature. Specifically, the input feature r is first transformed into a tensor of L×(T·H·W) through a linear layer φ(r), where T, H, and W represent the time axis, height dimension, and width dimension of the video feature, respectively. It is then transformed into a tensor of C×(T·H·W) through a linear layer and added to the original video feature to obtain a new feature representation f. Next, a convolutional neural network (CNN) is used to convert it into a feature representation of shape T×H×W×N, and the softmax function is used to select the most important information in each tag representation. Finally, the self-attention mechanism is used to model the relationship between different tags, and this feature representation is fused with the original video feature. The visual transformer trained by the model of the embodiment of the present invention can better adapt to the video text alignment task.

[0111] As a further optional implementation, performing visual-text alignment training on the query transformation model according to the target visual transformer to obtain a target query transformation model includes:

[0112] obtaining sample data, the sample data comprising a visual representation and a textual representation;

[0113] Extracting and processing the sample data by the target visual transformer to obtain a query representation;

[0114] Aligning the query representation with the text representation to obtain text similarity;

[0115] The parameters of the text transformer are updated according to the text similarity to obtain a target query transformation model.

[0116] In embodiments of the present invention, it is necessary to align the visual representation and the textual representation to maximize the mutual information between them. Specifically, this embodiment of the present invention accomplishes this task by learning the visual-textual similarity of positive sample pairs and comparing it with negative sample pairs. First, sample data is acquired, comprising a visual representation and a textual representation. The visual representation is extracted from video images and can be generated using the previously constructed visual encoder. The textual representation is typically obtained by inputting natural language text into a text transformer. A query is then used as a bridge between the visual and textual representations, and the query representation Z from the visual encoder is aligned with the output embedding t of the text transformer. Specifically, the sample data is extracted by the target visual transformer to obtain a query representation, which is then aligned with the textual representation to obtain textual similarity. Since the query representation Z contains multiple output embeddings, the pairwise similarity between each query output and t is first calculated, and the highest similarity is selected as the visual-textual similarity. To prevent information leakage, a unimodal self-attention mask is used to prevent the query and text from seeing each other. Finally, the parameters of the text transformer are updated according to the text similarity to obtain the target query transformation model. In this process, the visual transformer can be fixed and only the text transformer can be trained. The advantage of doing so is that the model parameters and the amount of computation can be reduced, and the trained visual transformer can be more fully utilized. Specifically, when the visual transformer is fixed and the visual encoder and q-former model are combined for visual-text alignment training, the trained visual transformer is used to extract the query representation Z, and then aligned with the output embedding t of the text transformer. The visual transformer here is fixed and no longer trained. Compared with training the visual transformer and the text transformer at the same time in an end-to-end model, this method can reduce the model parameters and the amount of computation, and can make better use of the trained transformation encoder.

[0117] In one feasible embodiment, a multimodal interaction model is generated for question-answering output by training a query transformation model through a linked language model through a fully connected layer. In the generative pre-training stage, a Q-Former (with a frozen image encoder) is connected to a frozen LLM to leverage the LLM's generative language capabilities. The query embedding Z is linearly projected to the same dimension as the LLM's text embedding using a fully connected layer (FC layer). The projected query embedding is then concatenated with the input text embedding as a soft visual cue to condition the LLM and provide it with the visual representation extracted by the Q-Former. Because the Q-Former has been pre-trained to extract language-related visual representations, it can effectively act as an information bottleneck, providing the most useful information to the LLM and removing irrelevant visual information. This reduces the burden of the LLM in learning visual-language alignment, thereby alleviating the forgetting problem in the model.

[0118] On the other hand, referring to FIG5 , an embodiment of the present invention further provides a multimodal interaction device based on an improved alignment method, the device comprising:

[0119] The first module 501 is used to obtain multimodal data;

[0120] The second module 502 is used to build a query transformation model, wherein the query transformation model includes a visual encoder, a visual transformer and a text transformer;

[0121] The third module 503 is used to improve the visual transformer according to the multi-stream feature extraction method to obtain a multi-scale visual transformer;

[0122] The fourth module 504 is configured to train the multi-scale visual transformer according to the joint text method to obtain a target visual transformer;

[0123] The fifth module 505 is configured to freeze the target visual transformer and perform visual-text alignment training on the query transformation model according to the target visual transformer to obtain a target query transformation model.

[0124] The sixth module 506 is configured to connect the target query transformation model to the language model to obtain a knowledge fusion multimodal interaction model;

[0125] The seventh module 507 is used to input the multimodal data into the knowledge fusion multimodal interaction model to obtain an interaction result.

[0126] 6 , an embodiment of the present invention further provides an electronic device, including a processor 601 and a memory 602 ; the memory is used to store programs; and the processor executes the programs to implement the method described above.

[0127] Corresponding to the method of FIG. 1 , an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0128] Embodiments of the present invention further disclose a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the method shown in FIG1 .

[0129] In summary, the embodiments of the present invention have the following advantages: The multimodal interaction model proposed in the embodiments of the present invention can utilize a multi-stream video encoder to process different types of information (such as RGB and optical flow) in video frames and feed this information into a query transformation model for cross-modal interaction. Furthermore, a multi-level alignment mechanism is introduced to effectively improve the multimodal alignment effect on videos. Finally, a language model is used for reasoning and answering, improving the multimodal interaction capabilities of videos.

[0130] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0131] Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise indicated, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It will also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, a person skilled in the art using ordinary skill will be able to implement the present invention set forth in the claims without undue experimentation. It will also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0132] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0133] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0134] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0135] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0136] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0137] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0138] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A knowledge fusion multimodal interaction method based on an improved alignment method, characterized in that: The multimodal interaction method comprises: Acquire multimodal data; Constructing a query transformation model, wherein the query transformation model includes a visual encoder, a visual transformer, and a text transformer; The visual transformer is improved according to a multi-stream feature extraction method to obtain a multi-scale visual transformer; Training the multi-scale visual transformer according to the joint text method to obtain a target visual transformer; Freezing the target visual transformer, and performing visual text alignment training on the query transformation model according to the target visual transformer to obtain a target query transformation model; Connecting the target query transformation model to the language model to obtain a knowledge fusion multimodal interaction model; The multimodal data is input into the knowledge fusion multimodal interaction model to obtain an interaction result.

2. The multimodal interaction method according to claim 1, characterized in that: The step of constructing the query transformation model includes: Freezing the visual encoder and connecting the frozen visual encoder to the visual converter; The visual transformer and the text transformer interact with each other through a shared self-attention layer to construct a query transformation model.

3. The multimodal interaction method according to claim 1, characterized in that: The visual transformer is improved according to the multi-stream feature extraction method to obtain a multi-scale visual transformer, including: Obtain multiple video streams through the visual encoder output; A residual network is added to the visual transformer to obtain a multi-scale visual transformer, wherein the residual network is used to perform multi-stream feature extraction processing on the multiple video streams to obtain multi-scale features, and the residual network includes a residual block, a spatial pyramid pooling layer and a fully connected layer.

4. The multimodal interaction method according to claim 3, characterized in that: The performing multi-stream feature extraction processing on the multiple video streams to obtain multi-scale features includes: Performing convolution and identity mapping processing on the multiple video streams respectively through the residual block to obtain a mapping vector set; Performing local information extraction processing on the mapping vector set at different scales through the spatial pyramid pooling layer to obtain a video stream feature vector set; Performing cascade processing on the video stream feature vector set to obtain a cascade data set; The cascade data set is mapped through the fully connected layer to obtain multi-scale features.

5. The multimodal interaction method according to claim 4, characterized in that: The training process of the multi-scale visual transformer according to the joint text method to obtain a target visual transformer includes: Acquire training data, wherein the training data includes the multi-scale features and text features; Performing cascade fusion processing on the training data to obtain an input feature matrix; Performing multi-head self-attention processing on the input feature matrix through a self-attention layer to obtain an output feature matrix; Performing feature extraction processing on the output feature matrix through a feed-forward neural network to obtain splicing features; Combining the word segmentation model with the concatenated features, multimodal fusion processing is performed to obtain fusion features; The fusion features are updated to training data, and the process returns to the step of performing cascade fusion processing on the training data to obtain an input feature matrix, until the number of cycles reaches a preset threshold, and a target visual transformer is obtained.

6. The multimodal interaction method according to claim 5, characterized in that: The combined word segmentation model performs multimodal fusion processing on the splicing features to obtain fusion features, including: Performing word segmentation processing on the training data through the word segmentation model to obtain word segmentation features; Projecting the word segmentation features through a convolutional neural network to obtain feature tags; Performing relationship modeling on the feature tags through an autonomous attention mechanism to obtain a feature representation; The feature representation is concatenated with the concatenated feature to obtain a fused feature.

7. The multimodal interaction method according to claim 1, characterized in that: The step of performing visual text alignment training processing on the query transformation model according to the target visual transformer to obtain the target query transformation model includes: Obtaining sample data, the sample data comprising a visual representation and a textual representation; Extracting and processing the sample data by the target visual transformer to obtain a query representation; Aligning the query representation with the text representation to obtain text similarity; The parameters of the text transformer are updated according to the text similarity to obtain a target query transformation model.

8. A multimodal interaction device based on an improved alignment method, characterized in that: The device comprises: The first module is used to obtain multimodal data; The second module is used to construct a query transformation model, wherein the query transformation model includes a visual encoder, a visual transformer and a text transformer; The third module is used to improve the visual transformer according to the multi-stream feature extraction method to obtain a multi-scale visual transformer; The fourth module is used to train the multi-scale visual transformer according to the joint text method to obtain a target visual transformer; A fifth module is used to freeze the target visual transformer and perform visual text alignment training on the query transformation model according to the target visual transformer to obtain a target query transformation model; The sixth module is used to connect the target query transformation model to the language model to obtain a knowledge fusion multimodal interaction model; The seventh module is used to input the multimodal data into the knowledge fusion multimodal interaction model to obtain an interaction result.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images