A knowledge fusion multi-modal interaction method and device based on an improved alignment method
By constructing a query transformation model and training visual-text alignment, the problem of low efficiency in multimodal data interaction in existing technologies is solved, and efficient multimodal interaction processing, especially video and text interaction, is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SUN YAT SEN UNIV
- Filing Date
- 2023-08-03
- Publication Date
- 2026-07-21
AI Technical Summary
Existing human-computer interaction methods mainly rely on text or voice, which cannot effectively process multimodal data such as images combined with text, resulting in poor interaction efficiency.
We employ a knowledge fusion multimodal interaction method based on an improved alignment approach. By constructing a query transformation model, including a visual encoder, a visual transformer, and a text transformer, we train the visual transformer using multi-stream feature extraction and joint text methods, perform visual-text alignment training, and connect it to a language model for multimodal data processing.
It improves the efficiency of multimodal interaction, effectively processes multimodal data, especially video and text interactions, and enhances multimodal alignment and interaction capabilities.
Smart Images
Figure CN117113270B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a knowledge fusion multimodal interaction method and apparatus based on an improved alignment method. Background Technology
[0002] In recent years, both computer vision and natural language processing have made rapid progress. However, many practical problems are inherently multimodal, meaning they involve several different forms of data simultaneously, such as images and text. Most human-computer interaction methods in related technologies simply rely on text or voice for interaction, failing to process multimodal data such as images combined with text, resulting in poor interaction efficiency. Therefore, the technical problems existing in these technologies urgently need to be solved. Summary of the Invention
[0003] In view of this, embodiments of the present invention provide a knowledge fusion multimodal interaction method and apparatus based on an improved alignment method to improve human-computer interaction efficiency.
[0004] On one hand, the present invention provides a knowledge fusion multimodal interaction method based on an improved alignment method, the multimodal interaction method comprising:
[0005] Acquire multimodal data;
[0006] Construct a query transformation model, which includes a visual encoder, a visual transformer, and a text transformer;
[0007] The visual transformer is improved based on the multi-stream feature extraction method to obtain a multi-scale visual transformer.
[0008] The multi-scale visual transformer is trained using a joint text method to obtain the target visual transformer.
[0009] The target visual transformer is frozen, and the query transformation model is trained by visual text alignment based on the target visual transformer to obtain the target query transformation model.
[0010] By connecting the target query transformation model to the language model, a knowledge fusion multimodal interaction model is obtained;
[0011] The multimodal data is input into the knowledge fusion multimodal interaction model to obtain the interaction results.
[0012] Optionally, constructing the query transformation model includes:
[0013] The visual encoder is frozen, and the frozen visual encoder is connected to the visual transformer.
[0014] The visual transformer and the text transformer interact through a shared self-attention layer to construct a query transformation model.
[0015] Optionally, the step of improving the visual transformer according to the multi-stream feature extraction method to obtain a multi-scale visual transformer includes:
[0016] Multiple video streams are obtained by outputting the visual encoder;
[0017] A residual network is added to the visual transformer to obtain a multi-scale visual transformer. The residual network is used to perform multi-stream feature extraction processing on the multiple video streams to obtain multi-scale features. The residual network includes residual blocks, spatial pyramid pooling layers, and fully connected layers.
[0018] Optionally, the step of performing multi-stream feature extraction processing on the multiple video streams to obtain multi-scale features includes:
[0019] The residual blocks are used to perform convolution and identity mapping on the multiple video streams respectively to obtain a set of mapping vectors;
[0020] The spatial pyramid pooling layer performs local information extraction processing on the mapping vector set at different scales to obtain a video stream feature vector set.
[0021] The video stream feature vector set is concatenated to obtain a concatenated dataset;
[0022] The cascaded dataset is mapped using the fully connected layer to obtain multi-scale features.
[0023] Optionally, the step of training the multi-scale visual transformer according to the joint text method to obtain the target visual transformer includes:
[0024] Acquire training data, which includes the multi-scale features and text features;
[0025] The training data is cascaded and fused to obtain the input feature matrix;
[0026] The input feature matrix is processed by a multi-head self-attention layer to obtain the output feature matrix;
[0027] The output feature matrix is processed by a feedforward neural network to extract features, resulting in concatenated features.
[0028] The concatenated features are then subjected to multimodal fusion processing using a word segmentation model to obtain fused features.
[0029] The fused features are updated to training data, and the process is returned to the step of cascading fusion of the training data to obtain the input feature matrix, until the number of iterations reaches a preset threshold, thus obtaining the target visual transformer.
[0030] Optionally, the step of combining the word segmentation model with the concatenated features to perform multimodal fusion processing to obtain fused features includes:
[0031] The training data is segmented using the segmentation model to obtain segmentation features;
[0032] The word segmentation features are projected using a convolutional neural network to obtain feature tags;
[0033] The feature labels are modeled using an autonomous attention mechanism to obtain feature representations.
[0034] The feature representation is combined with the spliced feature to obtain the fused feature.
[0035] Optionally, the step of performing visual text alignment training on the query transformation model based on the target visual transformer to obtain the target query transformation model includes:
[0036] Acquire sample data, which includes visual representation and text representation;
[0037] The target visual transformer is used to extract and process the sample data to obtain a query representation;
[0038] The query representation and the text representation are aligned to obtain the text similarity.
[0039] The parameters of the text transformer are updated based on the text similarity to obtain the target query transformation model.
[0040] On the other hand, embodiments of the present invention also provide a multimodal interaction device based on an improved alignment method, the device comprising:
[0041] The first module is used to acquire multimodal data;
[0042] The second module is used to construct a query transformation model, which includes a visual encoder, a visual transformer, and a text transformer.
[0043] The third module is used to improve the visual transformer according to the multi-stream feature extraction method to obtain a multi-scale visual transformer.
[0044] The fourth module is used to train the multi-scale visual transformer according to the joint text method to obtain the target visual transformer.
[0045] The fifth module is used to freeze the target visual transformer and perform visual text alignment training on the query transformation model based on the target visual transformer to obtain the target query transformation model.
[0046] The sixth module is used to connect the target query transformation model to the language model to obtain a knowledge fusion multimodal interaction model;
[0047] The seventh module is used to input the multimodal data into the knowledge fusion multimodal interaction model to obtain the interaction results.
[0048] On the other hand, embodiments of the present invention also disclose an electronic device, including a processor and a memory;
[0049] The memory is used to store programs;
[0050] The processor executes the program to implement the method described above.
[0051] On the other hand, embodiments of the present invention also disclose a computer-readable storage medium storing a program that is executed by a processor to implement the methods described above.
[0052] On the other hand, embodiments of the present invention also disclose a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.
[0053] Compared with existing technologies, the present invention, employing the above technical solutions, has the following technical effects: In this embodiment, the visual transformer is improved using a multi-stream feature extraction method to obtain a multi-scale visual transformer, capable of extracting different types of features from video frames; furthermore, the present invention trains the query transformation model using the target visual transformer for visual-text alignment, improving the multimodal alignment effect on the video; and moreover, by connecting the target query transformation model to a language model, a knowledge fusion multimodal interaction model is obtained, enhancing the multimodal interaction capability on the video. Attached Figure Description
[0054] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0055] Figure 1 This is a flowchart of a knowledge fusion multimodal interaction method based on an improved alignment method provided in an embodiment of this application;
[0056] Figure 2 This is a schematic diagram of the structure of a query transformation model provided in an embodiment of this application;
[0057] Figure 3 This is a schematic diagram of the structure of a multi-scale vision transformer provided in an embodiment of this application;
[0058] Figure 4 This is a training schematic diagram of a multi-scale visual transformer provided in an embodiment of this application;
[0059] Figure 5 This is a schematic diagram of the structure of a knowledge fusion multimodal interaction device based on an improved alignment method provided in an embodiment of this application;
[0060] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0061] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0062] This application provides a knowledge fusion multimodal interaction method. This interaction method can be applied to a terminal, a server, or software running on either a terminal or server. The terminal can be a tablet, laptop, desktop computer, etc., but is not limited to these. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.
[0063] Reference Figure 1 This invention provides a knowledge fusion multimodal interaction method based on an improved alignment method, the multimodal interaction method comprising:
[0064] S101. Acquire multimodal data;
[0065] S102. Construct a query transformation model, which includes a visual encoder, a visual transformer, and a text transformer.
[0066] S103. The visual transformer is improved according to the multi-stream feature extraction method to obtain a multi-scale visual transformer.
[0067] S104. The multi-scale visual transformer is trained according to the joint text method to obtain the target visual transformer.
[0068] S105. Freeze the target visual transformer and perform visual text alignment training on the query transformation model based on the target visual transformer to obtain the target query transformation model.
[0069] S106. Connect the target query transformation model to the language model to obtain a knowledge fusion multimodal interaction model;
[0070] S107. Input the multimodal data into the knowledge fusion multimodal interaction model to obtain the interaction result.
[0071] In this embodiment of the invention, the multimodal interaction method can be applied to a terminal for human-computer interaction. For example, multimodal data can be input via a target object on a computer, and the terminal can process the data using the multimodal interaction method of this embodiment, outputting corresponding interaction results, such as answers or suggestions. This embodiment first acquires multimodal data, including images, text, and video, which is obtained through upload or input by the target object. This embodiment primarily analyzes and processes the multimodal data uploaded by the target object using a knowledge fusion multimodal interaction model. Specifically, it constructs a query transformation model, which includes a visual encoder, a visual transformer, and a text transformer. To handle multi-stream data, this embodiment improves the visual transformer using a multi-stream feature extraction method to obtain a multi-scale visual transformer. This multi-scale visual transformer can simultaneously consider multiple different scales of video input in time and space. Then, the multi-scale visual transformer is trained using a joint text method to obtain a target visual transformer. The target visual transformer is then frozen, and the query transformation model is trained using visual-text alignment based on the target visual transformer to obtain the target query transformation model. In the visual-text alignment learning task, visual representations and text representations need to be aligned to maximize their mutual information. Since the target visual transformer is fixed and no longer trained, this method reduces model parameters and computational load compared to training both the visual and text transformers simultaneously in an end-to-end model, and allows for more efficient use of the already trained visual transformer. Finally, the target query transformation model is connected to a language model to obtain a knowledge fusion multimodal interaction model. By inputting the multimodal data into the knowledge fusion multimodal interaction model, interaction results are obtained. For example, a target object uploads a video to a terminal and inputs corresponding text content. The knowledge fusion multimodal interaction model in this embodiment analyzes the video and text content and outputs the corresponding interaction results.
[0072] It should be further noted that in various specific embodiments of this application, when processing data related to the identity or characteristics of the target object, such as the target object's information, behavioral data, historical data, and location information, is required, the target object's permission or consent will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require obtaining sensitive information about the target object, separate permission or consent from the target object will be obtained through pop-ups or redirection to a confirmation page. Only after obtaining the target object's separate permission or consent will the necessary target object-related data for the normal operation of the embodiments of this application be obtained.
[0073] As a further optional implementation, the construction of the query transformation model includes:
[0074] The visual encoder is frozen, and the frozen visual encoder is connected to the visual transformer.
[0075] The visual transformer and the text transformer interact through a shared self-attention layer to construct a query transformation model.
[0076] In this embodiment of the invention, reference is made to Figure 2 The query transformation model comprises a visual encoder and a Q-Former model. The Q-Former model is a trainable module used to bridge the gap between a frozen visual encoder and a fixed language model (LLM), capable of extracting a fixed number of output features from the visual encoder, independent of the input image resolution. The Q-Former consists of two transformer sub-modules: a visual transformer and a text transformer, sharing the same self-attention layer. One is an image transformer that interacts with the frozen visual encoder, used for visual feature extraction, and includes a self-attention layer, a cross-attention layer, and a feedforward layer. The visual encoder is frozen and then connected to the visual transformer. The other is a text transformer, which can function as both a text encoder and a text decoder, including a self-attention layer and a feedforward layer. The visual transformer and the text transformer interact through the shared self-attention layer to construct the query transformation model. This embodiment of the invention creates a set of learnable embeddings as the input query embeddings for the visual transformer. Query interactions occur through self-attention layers and with frozen image features via cross-attention layers (inserted into each transformer block). Depending on the pre-training task, different self-attention masks are used in this embodiment to control query text interactions. This bottleneck architecture, together with the pre-training objective of this embodiment, forces queries to extract the most relevant visual information to the text.
[0077] As a further optional implementation, the step of improving the visual transformer according to the multi-stream feature extraction method to obtain a multi-scale visual transformer includes:
[0078] Multiple video streams are obtained by outputting the visual encoder;
[0079] A residual network is added to the visual transformer to obtain a multi-scale visual transformer. The residual network is used to perform multi-stream feature extraction processing on the multiple video streams to obtain multi-scale features. The residual network includes residual blocks, spatial pyramid pooling layers, and fully connected layers.
[0080] In this embodiment of the invention, the visual transformer is improved. The construction process of this visual transformer is based on the idea of multi-scale feature extraction, which can simultaneously consider multiple different scales of video input in time and space. Specifically, this embodiment uses a multi-stream method, where each stream extracts features for different spatial and temporal scales. These features are then concatenated and processed together with the text / question input. The multi-scale visual transformer is obtained mainly by adding a residual network to the visual transformer. The residual network is used to perform multi-stream feature extraction processing on the multiple video streams to obtain multi-scale features. The residual network includes residual blocks, spatial pyramid pooling layers, and fully connected layers. Specifically, this embodiment uses the X3D network as the base model, which has performed well in many VideoQA tasks. (See reference...) Figure 3 In this embodiment of the invention, a residual network (ResNet) is used to extract features from each video stream, and three different scale video streams are used to obtain feature representations of multiple visual modalities.
[0081] As a further optional implementation, the step of performing multi-stream feature extraction processing on the multiple video streams to obtain multi-scale features includes:
[0082] The residual blocks are used to perform convolution and identity mapping on the multiple video streams respectively to obtain a set of mapping vectors;
[0083] The spatial pyramid pooling layer performs local information extraction processing on the mapping vector set at different scales to obtain a video stream feature vector set.
[0084] The video stream feature vector set is concatenated to obtain a concatenated dataset;
[0085] The cascaded dataset is mapped using the fully connected layer to obtain multi-scale features.
[0086] In this embodiment of the invention, a set of residual blocks is used, each block comprising a 3×3 convolutional kernel and an identity mapping. The residual blocks are used to perform convolution and identity mapping on the multiple video streams respectively, resulting in a set of mapping vectors. This process can be represented by the following formula:
[0087] f vi =ResNet(x vi )
[0088] Where x vi This represents the input of the i-th video stream, corresponding to multiple frames within that stream. After each residual block, a Spatial Pyramid Pooling layer is added. This layer extracts local information from the mapped vector set at different scales, resulting in a set of video stream feature vectors to capture local information at various scales. Finally, by concatenating the features generated from each stream, the set of video stream feature vectors is concatenated to obtain a concatenated dataset. A fully connected layer then maps these datasets into a fixed-length vector f. v ,Right now:
[0089] f v =FCN9concat(f v1 ,f v2 ,f v3 ))
[0090] Here, `concat` represents concatenating the three feature sets together, and `FCN` represents the fully connected layer used. The concatenated dataset is mapped using the fully connected layer to obtain multi-scale features, where f... v It is a multi-scale feature for the entire video segment.
[0091] As a further optional implementation, the step of training the multi-scale visual transformer according to the joint text method to obtain the target visual transformer includes:
[0092] Acquire training data, which includes the multi-scale features and text features;
[0093] The training data is cascaded and fused to obtain the input feature matrix;
[0094] The input feature matrix is processed by a multi-head self-attention layer to obtain the output feature matrix;
[0095] The output feature matrix is processed by a feedforward neural network to extract features, resulting in concatenated features.
[0096] The concatenated features are then subjected to multimodal fusion processing using a word segmentation model to obtain fused features.
[0097] The fused features are updated to training data, and the process is returned to the step of cascading fusion of the training data to obtain the input feature matrix, until the number of iterations reaches a preset threshold, thus obtaining the target visual transformer.
[0098] In this embodiment of the invention, reference is made to Figure 4 The multi-scale visual transformer is trained using a joint text method, and further optimized using the same method. First, training data is acquired, including the multi-scale features and text features. Then, the training data is cascaded and fused to obtain the input feature matrix, specifically the features f generated by the video encoder. v Embedding of text / question features f t By cascading operations, a (M+N)×D input feature matrix X0 is obtained:
[0099]
[0100] Where M represents the number of video features, N represents the number of text features, and D represents the dimension of the features.
[0101] Next, this embodiment of the invention uses a multi-layer Transformer model to further process the feature matrix X0. In the i-th layer Transformer, the input feature matrix X0 is processed... i 1. A self-attention mechanism is applied to capture the relationships between different features. The input feature matrix is processed by a multi-head self-attention layer to obtain the output feature matrix Z. i , can be represented as:
[0102] Z i =MultiHead(X i 1)
[0103] Here, MultiHead represents Multi-Head Attention, which divides the input matrix into several sub-matrices and applies an attention mechanism to each sub-matrix, thereby obtaining multiple feature maps. Dividing the input matrix into K sub-matrices, MultiHead can be represented as:
[0104] MultiHead(X)=concat(head1,…,head K )
[0105] head k This represents the feature map obtained after applying an attention mechanism to the k-th submatrix. Next, in this embodiment of the invention, feature extraction processing is performed on the output feature matrix using a feedforward neural network to obtain concatenated features. Specifically, a feedforward neural network is used to process the output feature matrix Z... i Further processing is performed to extract more useful information. The output of this feedforward neural network is H.i Then it can be expressed as:
[0106] H i =FFN(Z i )
[0107] Here, FFN represents a feedforward neural network, which consists of two fully connected layers, each containing an activation function and some regularization operations. Finally, in this embodiment of the invention, the concatenated features are subjected to multimodal fusion processing using a word segmentation model to obtain fused features. In each Transformer layer, informative tokens are generated using a word segmentation model (TokenLearner). In the i-th Transformer layer, the input feature matrix X is first... i 1. The tokens are passed to the TokenLearner in the video section and the text / question section respectively, thus obtaining a set of informative tokens for each section. and Then, these tokens are concatenated and further processed through a fully connected layer to obtain an informative token T with dimension D. i :
[0108]
[0109] FCN2 represents a network with two fully connected layers, used to map concatenated tokens to fixed-length vectors. Finally, in each Transformer layer, the informative token T is... i and the feature matrix H output by the previous Transformer layer i The features are then concatenated to obtain the fused features. The concatenation operation can be represented as:
[0110] X i =concat(H i ,T i )
[0111] In this embodiment of the invention, the fused features are updated as training data and input into the next layer of the Transformer for processing. This involves returning to the step of cascading fusion processing of the training data to obtain the input feature matrix, until the number of iterations reaches a preset threshold, resulting in the target visual transformer. Typically, the number of fine-tuning iterations, i.e., the preset threshold, can be between several hundred and several thousand.
[0112] As a further optional implementation, the step of combining the word segmentation model with the concatenated features to perform multimodal fusion processing to obtain fused features includes:
[0113] The training data is segmented using the segmentation model to obtain segmentation features;
[0114] The word segmentation features are projected using a convolutional neural network to obtain feature tags;
[0115] The feature labels are modeled using an autonomous attention mechanism to obtain feature representations.
[0116] The feature representation is combined with the spliced feature to obtain the fused feature.
[0117] In this embodiment of the invention, the feature representation of each modality is compressed by learning Tokenize (word segmentation) and converted into a set of meaningful token representations for better multimodal fusion. Specifically, the method uses a Token Learner model (word segmentation module) to learn the word segmentation of each modality and convert it into a token representation of the same dimension. The advantage of this is that it can greatly reduce the number of feature representations, thereby improving the efficiency and accuracy of the model. In addition, since TokenLearner is an adaptive mechanism, it can automatically select the most important information according to the input data, further improving the model's performance. For video input, multi-scale, multi-stream features f{vi} are used, where i is the index of the feature stream. First, the training data is segmented by the word segmentation model to obtain word segmentation features. Specifically, the text features of the segmented words are summed with the video features to obtain a comprehensive feature f = f_{vi} + r. Then, a convolutional neural network (CNN) is used to project it onto an N-dimensional token representation space, and a self-attention mechanism is used to model the relationship between different tokens to obtain feature representations. The feature representations are then concatenated with the concatenated features to obtain fused features. Specifically, the input feature r is first transformed into an L×(T·H·W) tensor through a linear layer φ(r), where T, H, and W represent the time axis, height dimension, and width dimension of the video feature, respectively. Then, it is transformed into a C×(T·H·W) tensor through another linear layer and added to the original video feature to obtain a new feature representation f. Next, a convolutional neural network (CNN) is used to transform it into a feature representation of shape T×H×W×N, and a softmax function is used to select the most important information in each token representation. Finally, a self-attention mechanism is used to model the relationship between different tokens, and this feature representation is fused with the original video feature. The visual transformer trained by the model in this embodiment of the invention is better adapted to video text alignment tasks.
[0118] As a further optional implementation, the step of performing visual text alignment training on the query transformation model based on the target visual transformer to obtain the target query transformation model includes:
[0119] Acquire sample data, which includes visual representation and text representation;
[0120] The target visual transformer is used to extract and process the sample data to obtain a query representation;
[0121] The query representation and the text representation are aligned to obtain the text similarity.
[0122] The parameters of the text transformer are updated based on the text similarity to obtain the target query transformation model.
[0123] In this embodiment of the invention, it is necessary to align the visual representation and the text representation to maximize their mutual information. Specifically, this embodiment accomplishes this task by learning the visual-text similarity of positive sample pairs and comparing it with negative sample pairs. First, sample data is acquired, including visual and text representations. The visual representation is features extracted from video images and can be accomplished using the previously constructed visual encoder. The text representation is typically obtained by inputting natural language text into a text transformer. A query is used as a bridge between the visual and text representations, aligning the query representation Z from the visual encoder with the output embedding t of the text transformer. Specifically, the sample data is processed by the target visual transformer to obtain the query representation, which is then aligned with the text representation to obtain the text similarity. Since the query representation Z contains multiple output embeddings, the pairwise similarity between each query output and t is calculated first, and then the highest similarity is selected as the visual-text similarity. To avoid information leakage, a unimodal self-attention mask is used to prevent the query and text from seeing each other. Finally, the parameters of the text transformer are updated based on the text similarity to obtain the target query transformation model. In this process, the visual transformer can be fixed, and only the text transformer can be trained. This reduces model parameters and computational cost, and allows for more efficient use of the already trained visual transformer. Specifically, in the visual-text alignment training with the visual encoder and q-former model fixed, the already trained visual transformer is used to extract the query representation Z, which is then aligned with the output embedding t of the text transformer. Here, the visual transformer is fixed and not trained further. Compared to training both the visual transformer and the text transformer simultaneously in an end-to-end model, this method reduces model parameters and computational cost, and allows for more efficient use of the already trained transformer encoder.
[0124] In one feasible implementation, a multimodal interaction model for question-answering output is generated by training a query transformation model through a linked language model via fully connected layers. During the generative pre-training phase, a Q-Former (with a frozen image encoder) is connected to a frozen LLM to leverage the LLM's generative language capabilities. The query embedding is Z-linearly projected onto the text embedding of the LLM using fully connected layers (FC layers) to have the same dimension. The projected query embedding is then concatenated with the input text embedding as a soft visual cue to condition the LLM and provide it with the visual representation extracted by the Q-Former. Because the Q-Former has been pre-trained to extract language-related visual representations, it can effectively act as an information bottleneck, providing the LLM with the most useful information and removing irrelevant visual information. This alleviates the burden on the LLM to learn visual-language alignment, thereby mitigating the forgetting problem in the model.
[0125] On the other hand, refer to Figure 5 This invention also provides a multimodal interaction device based on an improved alignment method, the device comprising:
[0126] The first module 501 is used to acquire multimodal data;
[0127] The second module 502 is used to construct a query transformation model, which includes a visual encoder, a visual transformer, and a text transformer.
[0128] The third module 503 is used to improve the visual transformer according to the multi-stream feature extraction method to obtain a multi-scale visual transformer.
[0129] The fourth module 504 is used to train the multi-scale visual transformer according to the joint text method to obtain the target visual transformer.
[0130] The fifth module 505 is used to freeze the target visual transformer and perform visual text alignment training on the query transformation model based on the target visual transformer to obtain the target query transformation model.
[0131] The sixth module 506 is used to connect the target query transformation model to the language model to obtain a knowledge fusion multimodal interaction model;
[0132] The seventh module 507 is used to input the multimodal data into the knowledge fusion multimodal interaction model to obtain the interaction result.
[0133] Reference Figure 6This invention also provides an electronic device, including a processor 601 and a memory 602; the memory is used to store a program; the processor executes the program to implement the method described above.
[0134] and Figure 1 Corresponding to the method described above, embodiments of the present invention also provide a computer-readable storage medium storing a program that is executed by a processor to implement the method described above.
[0135] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform... Figure 1 The method shown.
[0136] In summary, the embodiments of the present invention have the following advantages: The multimodal interaction model proposed in these embodiments can utilize a multi-stream video encoder to process different types of information (such as RGB, optical flow, etc.) in video frames and feed this information into a query transformation model for cross-modal interaction. Furthermore, a multi-level alignment mechanism is introduced, which can effectively improve the multimodal alignment effect on the video. Finally, a language model is used for reasoning and response, enhancing the multimodal interaction capability on the video.
[0137] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.
[0138] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.
[0139] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0140] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.
[0141] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.
[0142] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0143] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0144] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.
[0145] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.
Claims
1. A knowledge fusion multimodal interaction method based on an improved alignment method, characterized in that, The multimodal interaction method includes: Acquire multimodal data; Construct a query transformation model, which includes a visual encoder, a visual transformer, and a text transformer; The visual transformer is improved based on the multi-stream feature extraction method to obtain a multi-scale visual transformer. The multi-scale visual transformer is trained using a joint text method to obtain the target visual transformer. The target visual transformer is frozen, and the query transformation model is trained by visual text alignment based on the target visual transformer to obtain the target query transformation model. By connecting the target query transformation model to the language model, a knowledge fusion multimodal interaction model is obtained; The multimodal data is input into the knowledge fusion multimodal interaction model to obtain the interaction results; The improvement process of the visual transformer based on the multi-stream feature extraction method to obtain a multi-scale visual transformer includes: Multiple video streams are obtained by outputting the visual encoder; A residual network is added to the visual transformer to obtain a multi-scale visual transformer. The residual network is used to perform multi-stream feature extraction processing on the multiple video streams to obtain multi-scale features. The residual network includes residual blocks, spatial pyramid pooling layers and fully connected layers. The process of extracting multi-stream features from the multiple video streams to obtain multi-scale features includes: The residual blocks are used to perform convolution and identity mapping on the multiple video streams respectively to obtain a set of mapping vectors; The spatial pyramid pooling layer performs local information extraction processing on the mapping vector set at different scales to obtain a video stream feature vector set. The video stream feature vector set is concatenated to obtain a concatenated dataset; The cascaded dataset is mapped using the fully connected layer to obtain multi-scale features; The step of training the multi-scale visual transformer according to the joint text method to obtain the target visual transformer includes: Acquire training data, which includes the multi-scale features and text features; The training data is cascaded and fused to obtain the input feature matrix; The input feature matrix is processed by a multi-head self-attention layer to obtain the output feature matrix; The output feature matrix is processed by a feedforward neural network to extract features, resulting in concatenated features. The concatenated features are then subjected to multimodal fusion processing using a word segmentation model to obtain fused features. The fused features are updated to training data, and the process is returned to the step of cascading fusion of the training data to obtain the input feature matrix, until the number of iterations reaches a preset threshold, thus obtaining the target visual transformer.
2. The multimodal interaction method according to claim 1, characterized in that, The construction of the query transformation model includes: The visual encoder is frozen, and the frozen visual encoder is connected to the visual transformer. The visual transformer and the text transformer interact through a shared self-attention layer to construct a query transformation model.
3. The multimodal interaction method according to claim 1, characterized in that, The concatenated features are subjected to multimodal fusion processing using a word segmentation model to obtain fused features, including: The training data is segmented using the segmentation model to obtain segmentation features; The word segmentation features are projected using a convolutional neural network to obtain feature tags; The feature labels are modeled using an autonomous attention mechanism to obtain feature representations. The feature representation is combined with the spliced feature to obtain the fused feature.
4. The multimodal interaction method according to claim 1, characterized in that, The step of performing visual text alignment training on the query transformation model based on the target visual transformer to obtain the target query transformation model includes: Acquire sample data, which includes visual representation and text representation; The target visual transformer is used to extract and process the sample data to obtain a query representation; The query representation and the text representation are aligned to obtain the text similarity. The parameters of the text transformer are updated based on the text similarity to obtain the target query transformation model.
5. A multimodal interaction device based on an improved alignment method, characterized in that, The device includes: The first module is used to acquire multimodal data; The second module is used to construct a query transformation model, which includes a visual encoder, a visual transformer, and a text transformer. The third module is used to improve the visual transformer according to the multi-stream feature extraction method to obtain a multi-scale visual transformer. The fourth module is used to train the multi-scale visual transformer according to the joint text method to obtain the target visual transformer. The fifth module is used to freeze the target visual transformer and perform visual text alignment training on the query transformation model based on the target visual transformer to obtain the target query transformation model. The sixth module is used to connect the target query transformation model to the language model to obtain a knowledge fusion multimodal interaction model; The seventh module is used to input the multimodal data into the knowledge fusion multimodal interaction model to obtain the interaction results; The improvement process of the visual transformer based on the multi-stream feature extraction method to obtain a multi-scale visual transformer includes: Multiple video streams are obtained by outputting the visual encoder; A residual network is added to the visual transformer to obtain a multi-scale visual transformer. The residual network is used to perform multi-stream feature extraction processing on the multiple video streams to obtain multi-scale features. The residual network includes residual blocks, spatial pyramid pooling layers and fully connected layers. The process of extracting multi-stream features from the multiple video streams to obtain multi-scale features includes: The residual blocks are used to perform convolution and identity mapping on the multiple video streams respectively to obtain a set of mapping vectors; The spatial pyramid pooling layer performs local information extraction processing on the mapping vector set at different scales to obtain a video stream feature vector set. The video stream feature vector set is concatenated to obtain a concatenated dataset; The cascaded dataset is mapped using the fully connected layer to obtain multi-scale features; The step of training the multi-scale visual transformer according to the joint text method to obtain the target visual transformer includes: Acquire training data, which includes the multi-scale features and text features; The training data is cascaded and fused to obtain the input feature matrix; The input feature matrix is processed by a multi-head self-attention layer to obtain the output feature matrix; The output feature matrix is processed by a feedforward neural network to extract features, resulting in concatenated features. The concatenated features are then subjected to multimodal fusion processing using a word segmentation model to obtain fused features. The fused features are updated to training data, and the process is returned to the step of cascading fusion of the training data to obtain the input feature matrix, until the number of iterations reaches a preset threshold, thus obtaining the target visual transformer.
6. An electronic device, characterized in that, The electronic device includes a memory and a processor; The memory is used to store programs; The processor executes the program to implement the knowledge fusion multimodal interaction method as described in any one of claims 1 to 4.
7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the knowledge fusion multimodal interaction method as described in any one of claims 1 to 4.