Multi-modal feature generation method and device, electronic equipment and storage medium

Abstract

The invention relates to the field of multi-modal feature processing, in particular to a multi-modal feature generation method and device, electronic equipment and a storage medium. According to the method, the audio and image data are extracted from the original video and are respectively coded and processed, so that the data characteristics of different modalities can be fully mined, and the limitation that a single-modal model can only process one type of data is avoided. And meanwhile, a prompt text containing large language model task content and different modal feature association relationships is also obtained, so that a clear task direction can be provided for the model, and the model is guided to better fuse multi-modal information. Then different modal features and prompt texts are spliced to obtain multi-modal features, the multi-modal features serve as input of a large language model, the large language model can receive and process feature information of multiple types of data such as texts, images and audios at the same time, the defects of a single modal are overcome by fusing the different modal features, and the multi-modal features are obtained. Therefore, the accuracy of processing complex tasks is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of multimodal feature processing, and in particular to a method, device, electronic device and storage medium for generating multimodal features. Background Art

[0002] Artificial intelligence technology is developing rapidly, especially in the field of deep learning. In this context, multimodal fusion has gradually become a key technology for processing information in complex scenarios. In the past, traditional single-modal models were not very accurate in dealing with complex real-world tasks such as cross-media retrieval, intelligent dialogue systems, and sentiment analysis, because these tasks usually involve multiple different types of data such as text, images, and audio. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a method, device, electronic device and storage medium for generating multimodal features to solve the problem of low accuracy of traditional single modal models when processing complex tasks involving multiple types of data such as text, images, audio, etc.

[0004] In a first aspect, an embodiment of the present invention provides a method for generating a multimodal feature, the method comprising:

[0005] Obtain audio data and image data extracted from the original video;

[0006] Encoding the audio data to obtain hidden layer features of the audio data, and processing the image data to obtain compressed image features;

[0007] Acquire a prompt text, wherein the prompt text includes the task content of the large language model and the correlation relationship between the hidden layer feature and the compressed image feature;

[0008] The hidden layer features, the compressed image features and the prompt text are concatenated to obtain multimodal features.

[0009] Furthermore, encoding the audio data to obtain hidden layer features of the audio data includes:

[0010] Initializing the parameters of the audio encoding module using the weight parameters of the speech recognition model to obtain an initialized audio encoding module, wherein the audio encoding module includes a sampling submodule and a coding submodule;

[0011] Input the audio data into a sampling submodule in the initialized audio encoding module, downsample the audio data through the sampling submodule to obtain initial audio features, and transmit the initial audio features to the encoding submodule;

[0012] The initial audio features are gradually integrated through the encoding submodule to obtain hidden layer features at different levels.

[0013] Furthermore, the processing of the image data to obtain compressed image features includes:

[0014] Acquire a visual language model, wherein the visual language model includes a visual encoding module and a visual language adaptation module;

[0015] Inputting the image data into the visual encoding module, splitting the image into image blocks through the visual encoding module, generating an image feature sequence based on image features of the image blocks, and transmitting the image feature sequence to the visual language adaptation module;

[0016] The image feature sequence is compressed by the visual language adaptation module to obtain the compressed image feature.

[0017] Furthermore, the compressing the image feature sequence by the visual language adaptation module to obtain the compressed image feature includes:

[0018] Acquire a query vector, wherein the query vector is used to guide the visual language adaptation module to compress image features;

[0019] The image feature sequence and the query vector are used to construct an initial key-value pair through the visual language adaptation module, and a two-dimensional absolute position code of each image data is added to the initial key-value pair to obtain a target key-value pair, wherein the query vector in the key-value pair is a key name, and the image feature in the image feature sequence is a key value;

[0020] The image feature sequence in the target key-value pair is processed based on a cross-attention mechanism to obtain a compressed image feature of a fixed length.

[0021] Furthermore, the image feature sequence in the target key-value pair is processed based on the cross attention mechanism to obtain a compressed image feature of a fixed length, including:

[0022] Extracting image features contained in the image feature sequence from the target key-value pair;

[0023] Calculating the attention weight of the query vector and the image feature based on the dot product of each query vector and the image feature and the two-dimensional absolute position code;

[0024] Performing weighted summation on the image feature sequence according to the attention weight;

[0025] The above process is repeated until a compressed image feature of a fixed length is obtained.

[0026] Furthermore, the concatenation of the hidden layer features, the compressed image features and the prompt text to obtain multimodal features includes:

[0027] Determine the splicing order based on the task content;

[0028] The hidden layer features, the compressed image features and the text features corresponding to the prompt text are spliced ​​in the splicing order to obtain a splicing feature sequence, and the splicing feature sequence is used as the multimodal feature.

[0029] Furthermore, the method further comprises:

[0030] Obtaining an output result of the large language model based on the multimodal features;

[0031] Analyze the output results to obtain the importance of different modal features in the multimodal features;

[0032] The concatenation weights of the hidden layer features, the compressed image features and the text features are respectively configured according to the importance levels corresponding to the different modal features;

[0033] Based on the hidden layer features, the splicing weights corresponding to the compressed image features and the text features are spliced ​​to obtain updated multimodal features.

[0034] In a second aspect, an embodiment of the present invention provides a device for generating a multimodal feature, the device comprising:

[0035] An extraction module, used for obtaining audio data and image data extracted from the original video;

[0036] A processing module, used for encoding the audio data to obtain hidden layer features of the audio data, and processing the image data to obtain compressed image features;

[0037] An acquisition module, used for acquiring a prompt text, wherein the prompt text includes the task content of the large language model and the correlation relationship between the hidden layer feature and the compressed image feature;

[0038] The splicing module is used to splice the hidden layer features, the compressed image features and the prompt text to obtain multimodal features.

[0039] In a third aspect, an embodiment of the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, computer instructions being stored in the memory, and the processor executing the method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0040] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method of the first aspect or any corresponding embodiment thereof.

[0041] The method provided in the embodiment of the present application can fully exploit the data features of different modalities by extracting audio and image data from the original video and encoding and processing them separately, thereby avoiding the limitation that a single modal model can only process one type of data. At the same time, a prompt text containing the task content of the large language model and the correlation between different modal features is also obtained, which helps to provide the model with a clear task direction and guide it to better integrate multimodal information. Then, the multimodal features are obtained by splicing different modal features and prompt texts, and using them as the input of the large language model, so that the large language model can simultaneously receive and process feature information from multiple types of data such as text, images, and audio, and make up for the shortcomings of a single modality by fusing different modal features, thereby effectively improving the accuracy of processing complex tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 is a flowchart of a method for generating a multimodal feature according to some embodiments of the present invention;

[0044] Figure 2 is a schematic diagram of a process of extracting hidden layer features according to some embodiments of the present invention;

[0045] Figure 3 is a schematic diagram of a process of extracting compressed image features according to some embodiments of the present invention;

[0046] Figure 4 is a flowchart of a method for generating a multimodal feature according to some embodiments of the present invention;

[0047] Figure 5 is a structural block diagram of a method and device for generating multimodal features according to an embodiment of the present invention;

[0048] Figure 6 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0050] According to an embodiment of the present invention, a method, apparatus, electronic device and storage medium for generating multimodal features are provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0051] In this embodiment, a method for generating a multimodal feature is provided. Figure 1 is a flow chart of a method for generating a multimodal feature according to an embodiment of the present invention, such as Figure 1 As shown, the process includes the following steps:

[0052] Step S101, obtaining audio data and image data extracted from the original video.

[0053] In an embodiment of the present application, the method for extracting audio data can be to open the original video file using a video processing tool. Call the audio extraction function of the tool to separate the audio data in the video and save it as a specific audio format file. The method for extracting image data of a preset number of frames in the original video can be to open the original video using a video processing tool. Determine the value of the preset frame interval, for example, extract one frame of image every 10 frames. Starting from the first frame of the video, read the video frame by frame. Record the number of frames currently read. When the current number of frames is an integer multiple of the preset frame interval, save the frame image, continue to read the next frame, and repeat the above process until the end of the video, and you can get the image data of the preset number of frames.

[0054] Step S102, encoding the audio data to obtain hidden layer features of the audio data, and processing the image data to obtain compressed image features.

[0055] In the embodiment of the present application, encoding the audio data to obtain the hidden layer features of the audio data includes the following steps A1-A3:

[0056] Step A1, using the weight parameters of the speech recognition model to initialize the parameters of the audio encoding module to obtain an initialized audio encoding module, wherein the audio encoding module includes a sampling submodule and a coding submodule.

[0057] Specifically, obtain the weight parameter file of a pre-trained speech recognition model (such as Whisper-large-v3). Determine the structure of the audio encoding module, such as Figure 2 As shown, the audio encoding module includes a sampling submodule and an encoding submodule. For example, the sampling submodule can be a two-layer convolutional downsampling structure, and the encoding submodule can be a 32-layer Transformer encoder.

[0058] Load the weight parameters of the pre-trained speech recognition model. Map the loaded weight parameters to the corresponding parts of the audio encoding module. For the sampling submodule and encoding submodule in the audio encoding module, match the parameters according to their structure and the corresponding parts of the pre-trained model.

[0059] If the structure of the audio encoding module is not completely consistent with the pre-trained model in some parts, appropriate adjustments and mappings can be made according to the specific situation. For example, if the parameter dimensions of the convolutional layer of the sampling submodule are not exactly the same as the corresponding parts in the pre-trained model, the shape of the parameters can be adjusted or appropriate cropping and padding can be performed. For the Transformerencoder in the encoding submodule, the corresponding parameters in the pre-trained model can be copied to the Transformerencoder of the audio encoding module to ensure the correct initialization of the parameters, and finally obtain the initialized audio encoding module.

[0060] It should be noted that the pre-trained speech recognition model (such as Whisper-large-v3) has learned rich audio feature representations on large-scale data. Using its weight parameters for initialization can make the new audio encoding part at a relatively good starting point at the beginning of training, without having to start learning from a completely random initialization state. The time required for model convergence and the amount of training data are reduced, thereby accelerating the entire training process. In addition, the speech recognition model (such as Whisper-large-v3) has mastered some common patterns and features in audio data after training with a large amount of data. By inheriting its weight parameters, the new audio encoding part can directly use this prior knowledge to better understand and process the input audio data. Since the speech recognition model (such as Whisper-large-v3) is trained on different types of audio data, its weight parameters have a certain generalization ability. Using these parameters to initialize the new audio encoding part can help the new model better adapt to a variety of different audio inputs and improve the performance and generalization ability of the model.

[0061] Step A2: input the audio data into the sampling submodule in the initialized audio encoding module, downsample the audio data through the sampling submodule to obtain initial audio features, and transmit the initial audio features to the encoding submodule.

[0062] Specifically, the audio data is input to the sampling submodule of the audio encoding module. The sampling submodule is a two-layer convolution downsampling structure, and the audio data can be downsampled through the two layers of convolution operations in sequence. For example, the first layer of convolution can perform preliminary feature extraction and dimensionality reduction on the audio data, and the second layer of convolution can further downsample and extract more abstract features. After two layers of convolution downsampling, the initial audio features are obtained. The initial audio features are then transmitted to the encoding submodule. The initial audio features can be used as input to the encoding submodule, or passed to the encoding submodule through a suitable data connection method.

[0063] Step A3: The initial audio features are gradually integrated through the encoding submodule to obtain hidden layer features at different levels.

[0064] Specifically, since the encoding submodule is a 32-layer Transformer encoder, each layer can further process and integrate the input audio features. For example, the first layer can perform local context modeling on the initial audio features, and subsequent layers can gradually expand the context range and extract more advanced feature representations. In the processing of each layer, components such as multi-head attention mechanism and feedforward neural network are used to transform and integrate the audio features. The multi-head attention mechanism allows the model to focus on audio features at different positions, and the feedforward neural network can perform nonlinear transformations on the features. As the encoding submodule is processed layer by layer, hidden layer features at different levels are obtained. These hidden layer features contain information about the audio data at different levels of abstraction, which can be used for subsequent fusion with visual information.

[0065] Through the above steps, the audio data can be input into the initialized audio encoding module to obtain hidden layer features at different levels.

[0066] In the embodiment of the present application, the image data is processed to obtain compressed image features, including the following steps B1-B3:

[0067] Step B1, obtaining a visual language model, wherein the visual language model includes a visual encoding module and a visual language adaptation module.

[0068] Specifically, the visual language model includes: a visual encoding module (using the VisionTransformer (ViT) architecture and initialized to Openclip's ViT-bigG pre-trained weights) and a visual language adaptation module (including a randomly initialized single-layer cross-attention module).

[0069] Step B2: input the image data into the visual coding module, split the image into image blocks through the visual coding module, generate an image feature sequence based on the image features of the image blocks, and transmit the image feature sequence to the visual language adaptation module.

[0070] Specifically, Figure 3 As shown, the image data of the preset number of frames is input into the visual encoding module. The visual encoding module adopts the VisionTransformer (ViT) architecture, and first splits the input image into image blocks with a preset stride. For example, if the preset stride is 14, the image is divided into image blocks of equal size according to certain rules, and the stride between each image block is 14. Secondly, for each image block, the image features are extracted. Components such as the self-attention mechanism and feedforward neural network in ViT can be used to extract the feature representation of the image block. The features of all image blocks are combined to generate an image feature sequence. The image feature sequence is then transmitted to the visual language adaptation module.

[0071] Step B3: compress the image feature sequence through the visual language adaptation module to obtain compressed image features.

[0072] In the embodiment of the present application, the image feature sequence is compressed by the visual language adaptation module to obtain compressed image features, including the following steps B301-B303:

[0073] Step B301, obtaining a query vector, wherein the query vector is used to guide the visual language adaptation module to compress image features.

[0074] Specifically, the query vector is a set of trainable vectors (embeddings). During the model initialization phase, these vectors can be randomly initialized, and their dimensions and number can be determined based on the specific task requirements and model design. For example, if you are dealing with an image classification task, the dimension of the query vector can be selected based on the dimension of the image features and the complexity of the task. In general, higher-dimensional query vectors can capture more complex feature relationships, but they also increase computational costs.

[0075] It should be noted that the query vectors play a guiding role in the cross-attention operation. They interact with the image features and determine which parts of the image features are more important for the compression process by calculating the attention weights. In addition, the query vector can highlight those image features that are relevant to a specific task. During the compression process, image features that are highly correlated with the query vector will be given higher weights, thus occupying a more important position in the compressed features.

[0076] Step B302, construct an initial key-value pair with the image feature sequence and the query vector through the visual language adaptation module, and add the two-dimensional absolute position code of each image data to the initial key-value pair to obtain a target key-value pair, wherein the query vector in the key-value pair is the key name, and the image feature in the image feature sequence is the key value.

[0077] Specifically, first create a data structure to store key-value pairs, and add the query vectors as key names to the data structure one by one. For each image feature vector in the image feature sequence, add it to the data structure as the key value of the corresponding query vector, thereby constructing an initial key-value pair. Secondly, for each image data, calculate the two-dimensional absolute position code according to its position in the image. Then, traverse each key-value pair in the initial key-value pair. For each image feature vector (key value), add its corresponding two-dimensional absolute position code to update the initial key-value pair so that the new key value contains the image feature vector and position code information, and obtain the target key-value pair.

[0078] Step B303, processing the image feature sequence in the target key-value pair based on the cross-attention mechanism to obtain a compressed image feature of a fixed length.

[0079] Specifically, the image feature sequence in the target key-value pair is processed based on the cross-attention mechanism to obtain a compressed image feature of a fixed length, including: extracting the image features contained in the image feature sequence from the target key-value pair; calculating the attention weights of the query vector and the image feature based on the dot product of each query vector and the image feature and the two-dimensional absolute position encoding; performing weighted summation on the image feature sequence according to the attention weight; and repeating the above process until a compressed image feature of a fixed length is obtained.

[0080] First, traverse the target key-value pairs and extract each image feature from the key value. Second, traverse each image feature and calculate the dot product between the query vector and the current image feature. You can use the vector dot product calculation formula, which is to multiply the query vector and the corresponding elements of the image feature and then sum them. At the same time, combine the two-dimensional absolute position encoding to calculate the attention weight. For example, after the dot product calculation, adjust the result according to the position encoding. And use the softmax function to normalize the scaled dot product result to obtain the attention weight of the query vector and the current image feature.

[0081] Then for each query vector: perform weighted summation of the image features in the image feature sequence according to the calculated attention weights. That is, multiply each image feature by its corresponding attention weight, and then add up all the results. The result is a feature vector after weighted summation. Repeat the above process to continuously update the attention weights and weighted summation results.

[0082] Get a preset number of iterations or other stop conditions, for example, stop the iteration when the length of the compressed feature vector reaches a fixed length (such as 256). In each iteration, the query vector, image feature sequence or two-dimensional absolute position encoding can be adjusted as needed to optimize the compression effect. Finally, a fixed-length compressed image feature is obtained, that is, a feature vector after multiple iterations and weighted summation, which retains important information in the image feature sequence and reduces the amount of data through compression.

[0083] Step S103, obtaining a prompt text, wherein the prompt text includes the task content of the large language model and the correlation relationship between the hidden layer features and the compressed image features.

[0084] In an embodiment of the present application, a task description is first obtained, which includes specific tasks to be performed by the large language model, such as video content description, question and answer, sentiment analysis, etc. The correlation between the hidden layer features and the compressed image features is analyzed, such as from the perspectives of consistency of content themes, matching of sentiment tendencies, co-occurrence of specific objects, etc. In addition, the correlation between the hidden layer features and the compressed image features can also be obtained by annotation by annotators. Finally, a prompt text is constructed based on the task content of the large language model and the correlation between the hidden layer features and the compressed image features.

[0085] Step S104, concatenating hidden layer features, compressed image features and prompt text to obtain multimodal features.

[0086] In an embodiment of the present application, hidden layer features, compressed image features and prompt text are spliced ​​to obtain multimodal features, including: determining a splicing order based on the task content; splicing hidden layer features, compressed image features and text features corresponding to the prompt text in the splicing order to obtain a spliced ​​feature sequence, and using the spliced ​​feature sequence as a multimodal feature.

[0087] Specifically, first, analyze the specific content of the task. For example, if the task is to describe the content of the video, it may be more inclined to splice the image features first, so that the model has a preliminary understanding of the visual information, then splice the audio features to supplement the sound information, and finally splice the prompt text to clarify the specific description focus and direction. Secondly, consider the importance of different modal features to the task. If the task is highly dependent on audio information, such as music recognition tasks, the hidden layer features of the audio will be spliced ​​first. Then, determine the specific splicing order based on the task content. Assume that the determined order is compressed image features, hidden layer features, and text features corresponding to the prompt text.

[0088] The concatenation process is as follows: Create an empty list or array to store the concatenated feature sequence. First, add the compressed image feature to the list. If the compressed image feature is a vector sequence, simply add it element by element to the list. Next, add the hidden layer feature. Similarly, if the hidden layer feature is a vector sequence, add its elements to the list in order. Finally, add the text feature corresponding to the prompt text. After the concatenation is completed, the resulting list or array is the concatenated feature sequence, which is used as a multimodal feature.

[0089] In an embodiment of the present application, after the multimodal features are input into the large language model, a question-answering system can be constructed based on the output of the large language model, allowing users to ask questions about the video content and obtain accurate answers. For example, users can ask who a specific person in the video is, why an event occurred, etc. Alternatively, the output of the large language model can be translated into multiple languages ​​to expand the audience range of the video. Alternatively, the emotional tendencies conveyed by the video, such as joy, sadness, tension, etc., can be analyzed, and classified or recommended based on the results of the emotional analysis.

[0090] The method provided in the embodiment of the present application can fully exploit the data features of different modalities by extracting audio and image data from the original video and encoding and processing them separately, thereby avoiding the limitation that a single modality model can only process one type of data. At the same time, a prompt text containing the task content of the large language model and the correlation between different modality features is also obtained, which helps to provide the model with a clear task direction and guide it to better integrate multimodal information. Then, multimodal features are obtained by splicing different modal features and prompt texts, and using them as input to the large language model, so that the large language model can simultaneously receive and process feature information from multiple types of data such as text, images, and audio, and make up for the shortcomings of a single modality by integrating the advantages of different modalities, thereby effectively improving the accuracy of processing complex tasks.

[0091] Figure 4 is a flow chart of a method for generating a multimodal feature according to an embodiment of the present invention, such as Figure 4 As shown, the process includes the following steps: obtaining the output result of the large language model based on multimodal features;

[0092] Step S201, analyzing the output results to obtain the importance levels corresponding to different modal features in the multimodal features.

[0093] In an embodiment of the present application, multimodal features are input into a large language model and an output result is obtained. This output result can be a description of the video content, an answer to a question, etc. A variety of methods can be used to analyze the importance of different modal features. For example, feature importance analysis algorithms can be used, such as gradient-based methods, random forest feature importance, etc. If the large language model is interpretable, the importance information of different modal features can be directly extracted from the interpretation of the model.

[0094] For example, using a gradient-based approach, we can calculate the gradient of the large language model output for each modal feature. Modal features with larger gradients usually have a greater impact on the output results, so they can be considered to be more important.

[0095] Step S202, configuring the concatenation weights of hidden layer features, compressed image features, and text features respectively according to the importance levels corresponding to different modal features.

[0096] In the embodiment of the present application, the weight distribution strategy is determined: different weight distribution strategies can be adopted according to the importance obtained in step S201. The importance can be directly used as the splicing weight, or normalized as the weight.

[0097] Specifically, using importance as weight, assuming that the importance of hidden layer features is w1, the importance of compressed image features is w2, and the importance of text features is w3, then their corresponding splicing weights are w1, w2, and w3 respectively. Alternatively, normalization can be performed by first calculating the sum of the importance of all modal features S = w1 + w2 + w3, and then dividing each importance by the sum to obtain the normalized weight. For example, the splicing weight of hidden layer features is w1 / S, the splicing weight of compressed image features is w2 / S, and the splicing weight of text features is w3 / S.

[0098] Step S203, performing splicing based on the splicing weights corresponding to the hidden layer features, the compressed image features and the text features, to obtain updated multimodal features.

[0099] In an embodiment of the present application, an empty list or array is created to store the updated multimodal features. For hidden layer features, each element is multiplied by the corresponding splicing weight (or normalized weight) and then added to the list one by one. For compressed image features, each element is also multiplied by the corresponding splicing weight (or normalized weight) and added to the list. For text features, each element is multiplied by the corresponding splicing weight (or normalized weight) and added to the list. After splicing is completed, the resulting list or array is the updated multimodal feature. It can be used for subsequent analysis, processing, or re-entered into the large language model for further optimization.

[0100] The method provided in the embodiment of the present application determines the importance of different modal features by analyzing the output results of a large language model, and can dynamically adjust the splicing weight according to specific tasks and input data. The model can be better adapted to different types of video content and task requirements, thereby improving the adaptability of the model. Different videos may have different importance in audio, image, and text. For example, in a video dominated by music, audio features may be more critical; while in a video dominated by visual effects, image features may be more important. The embodiment of the present application can highlight the features of key modalities according to actual conditions, thereby increasing the model's attention to key information, and thereby improving the accuracy of the output results.

[0101] In addition, allocating splicing weights according to their importance can avoid giving too much computing resources to unimportant modal features. This can make more effective use of resources and improve computing efficiency under limited computing resources. For less important modal features, reducing their splicing weights can reduce their impact on the final result, thereby reducing possible redundant information.

[0102] In the present embodiment, a device for generating multimodal features is also provided, and the device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0103] This embodiment provides a device for generating multimodal features, such as Figure 5 As shown, including:

[0104] Extraction module 501, used to obtain audio data and image data extracted from the original video;

[0105] A processing module 502 is used to encode the audio data to obtain hidden layer features of the audio data, and to process the image data to obtain compressed image features;

[0106] An acquisition module 503 is used to acquire a prompt text, wherein the prompt text includes the task content of the large language model and the correlation between the hidden layer features and the compressed image features;

[0107] The splicing module 504 is used to splice the hidden layer features, the compressed image features and the prompt text to obtain multimodal features.

[0108] In an embodiment of the present application, the processing module 502 is used to initialize the parameters of the audio encoding module using the weight parameters of the speech recognition model to obtain an initialized audio encoding module, wherein the audio encoding module includes a sampling submodule and an encoding submodule; input the audio data into the sampling submodule in the initialized audio encoding module, downsample the audio data through the sampling submodule to obtain initial audio features, and transmit the initial audio features to the encoding submodule; gradually integrate the initial audio features through the encoding submodule to obtain hidden layer features at different levels.

[0109] In an embodiment of the present application, the processing module 502 is used to obtain a visual language model, wherein the visual language model includes a visual encoding module and a visual language adaptation module; input image data into the visual encoding module, split the image into image blocks through the visual encoding module, generate an image feature sequence based on the image features of the image blocks, and transmit the image feature sequence to the visual language adaptation module; compress the image feature sequence through the visual language adaptation module to obtain compressed image features.

[0110] In an embodiment of the present application, a processing module 502 is used to obtain a query vector, wherein the query vector is used to guide a visual language adaptation module to compress image features; an initial key-value pair is constructed using the image feature sequence and the query vector through the visual language adaptation module, and a two-dimensional absolute position encoding of each image data is added to the initial key-value pair to obtain a target key-value pair, wherein the query vector in the key-value pair is the key name, and the image feature in the image feature sequence is the key value; the image feature sequence in the target key-value pair is processed based on a cross-attention mechanism to obtain a compressed image feature of a fixed length.

[0111] In an embodiment of the present application, the processing module 502 is used to extract the image features contained in the image feature sequence from the target key-value pair; based on the dot product of each query vector and the image feature and the two-dimensional absolute position encoding, calculate the attention weight of the query vector and the image feature; perform weighted summation on the image feature sequence according to the attention weight; and repeat the above process until a compressed image feature of a fixed length is obtained.

[0112] In an embodiment of the present application, the splicing module 504 is used to determine the splicing order based on the task content; splice the hidden layer features, compressed image features and text features corresponding to the prompt text according to the splicing order to obtain a splicing feature sequence, and use the splicing feature sequence as a multimodal feature.

[0113] In an embodiment of the present application, the device also includes: an updating module, which is used to obtain the output results of the large language model based on the multimodal features; analyze the output results to obtain the importance levels corresponding to different modal features in the multimodal features; use the importance levels corresponding to different modal features to respectively configure the splicing weights of hidden layer features, compressed image features, and text features; splice based on the splicing weights corresponding to the hidden layer features, compressed image features, and text features, to obtain updated multimodal features.

[0114] See also Figure 6 , Figure 6 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 6 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 10 is taken as an example.

[0115] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0116] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0117] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of a computer device based on the presentation of a small program landing page, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0118] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0119] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0120] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0121] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for generating multimodal features, characterized in that: The method comprises: Obtain audio data and image data extracted from the original video; Encoding the audio data to obtain hidden layer features of the audio data, and processing the image data to obtain compressed image features; Acquire a prompt text, wherein the prompt text includes the task content of the large language model and the correlation relationship between the hidden layer feature and the compressed image feature; The hidden layer features, the compressed image features and the prompt text are concatenated to obtain multimodal features.

2. The method according to claim 1, characterized in that The step of encoding the audio data to obtain hidden layer features of the audio data includes: Initializing the parameters of the audio encoding module using the weight parameters of the speech recognition model to obtain an initialized audio encoding module, wherein the audio encoding module includes a sampling submodule and a coding submodule; Input the audio data into a sampling submodule in the initialized audio encoding module, downsample the audio data through the sampling submodule to obtain initial audio features, and transmit the initial audio features to the encoding submodule; The initial audio features are gradually integrated through the encoding submodule to obtain hidden layer features at different levels.

3. The method according to claim 1, characterized in that The step of processing the image data to obtain compressed image features includes: Acquire a visual language model, wherein the visual language model includes a visual encoding module and a visual language adaptation module; Inputting the image data into the visual encoding module, splitting the image into image blocks through the visual encoding module, generating an image feature sequence based on image features of the image blocks, and transmitting the image feature sequence to the visual language adaptation module; The image feature sequence is compressed by the visual language adaptation module to obtain the compressed image feature.

4. The method according to claim 3, characterized in that The step of compressing the image feature sequence by the visual language adaptation module to obtain the compressed image feature comprises: Acquire a query vector, wherein the query vector is used to guide the visual language adaptation module to compress image features; The image feature sequence and the query vector are used to construct an initial key-value pair through the visual language adaptation module, and a two-dimensional absolute position code of each image data is added to the initial key-value pair to obtain a target key-value pair, wherein the query vector in the key-value pair is a key name, and the image feature in the image feature sequence is a key value; The image feature sequence in the target key-value pair is processed based on a cross-attention mechanism to obtain a compressed image feature of a fixed length.

5. The method according to claim 4, characterized in that The method of processing the image feature sequence in the target key-value pair based on the cross attention mechanism to obtain a compressed image feature of a fixed length includes: Extracting image features contained in the image feature sequence from the target key-value pair; Calculating the attention weight of the query vector and the image feature based on the dot product of each query vector and the image feature and the two-dimensional absolute position code; Performing weighted summation on the image feature sequence according to the attention weight; The above process is repeated until a compressed image feature of a fixed length is obtained.

6. The method according to claim 1, characterized in that The step of concatenating the hidden layer features, the compressed image features and the prompt text to obtain a multimodal feature includes: Determine the splicing order based on the task content; The hidden layer features, the compressed image features and the text features corresponding to the prompt text are spliced ​​in the splicing order to obtain a splicing feature sequence, and the splicing feature sequence is used as the multimodal feature.

7. The method according to claim 1, characterized in that The method further comprises: Obtaining an output result of the large language model based on the multimodal features; Analyze the output results to obtain the importance of different modal features in the multimodal features; The concatenation weights of the hidden layer features, the compressed image features and the text features are respectively configured according to the importance levels corresponding to the different modal features; Based on the hidden layer features, the splicing weights corresponding to the compressed image features and the text features are spliced ​​to obtain updated multimodal features.

8. A device for generating multimodal features, characterized in that: The device comprises: An extraction module, used for obtaining audio data and image data extracted from the original video; A processing module, used for encoding the audio data to obtain hidden layer features of the audio data, and processing the image data to obtain compressed image features; An acquisition module, used for acquiring a prompt text, wherein the prompt text includes the task content of the large language model and the correlation relationship between the hidden layer feature and the compressed image feature; The splicing module is used to splice the hidden layer features, the compressed image features and the prompt text to obtain multimodal features.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images