Multi-modal data processing method and device based on unified representation model, equipment and medium

Abstract

The invention relates to the field of artificial intelligence, and provides a multi-modal data processing method based on a unified representation model, and the method comprises the steps: obtaining multi-modal feature data, processing the multi-modal feature data in a first attention-based fusion layer, obtaining initial fusion feature data, carrying out the secondary fusion of the initial fusion feature data, and obtaining a multi-modal feature data fusion layer; calculating the similarity between the secondary fusion feature data and the initial fusion feature data, generating a second attention weight based on the similarity, weighting the initial fusion feature data based on the second attention weight, and obtaining the weighted initial fusion feature data; and finally, fusing the weighted initial fusion feature data with the secondary fusion feature data to generate a unified multi-modal representation of the multi-modal data. According to the method, the rich information of the multi-modal data is fully utilized, and the information fusion efficiency is improved, so that the accuracy of the model in AI subtitle generation and image-text generation application is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data processing method, device, equipment and medium based on a unified representation model. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, multimodal learning has performed well in many fields (such as computer vision, natural language processing, and speech recognition). Multimodal learning models can simultaneously process and fuse data from different modalities (such as images, text, audio, etc.), thereby improving the performance and application effect of the model. However, existing multimodal learning methods still face many challenges in practical applications, especially in terms of information fusion efficiency, computing resource requirements, and overall model performance.

[0003] Existing multimodal data processing methods usually directly process each original modal data independently and then simply fuse the independently processed multimodal data at the high level of the network model. For example, in image and text matching, image data and text data are respectively extracted through convolutional neural networks and pre-trained language models (such as BERT language models), and then these features are simply concatenated or simply fused through a fully connected layer.

[0004] Therefore, this method suffers from the problem of low information fusion efficiency, which leads to low training and inference efficiency.

[0005] Application Contents

[0006] The main purpose of the embodiments of the present application is to provide a data processing method, device, equipment and medium based on a unified representation model, aiming to solve at least one of the problems existing in the multimodal learning methods in the related technologies in practical applications, such as low information fusion efficiency, low training and reasoning efficiency, insufficient generalization ability and ability to adapt to different tasks, etc.

[0007] In a first aspect, an embodiment of the present application provides a multimodal data processing method based on a unified representation model, the method comprising: obtaining multimodal feature data; in a first attention-based fusion layer, converting the multimodal feature data into a query vector, a key vector and a value vector respectively; calculating a first attention weight between the query vector and the key vector; weighting the value vector based on the first attention weight to obtain a weighted value vector; initially fusing the weighted value vector with the query vector to obtain initial fused feature data; in a fully connected fusion layer, fusing the initial fused feature data for a second time to form secondary fused feature data; in a second attention-based fusion layer, calculating the similarity between the secondary fused feature data and the initial fused feature data; generating a second attention weight based on the similarity; weighting the initial fused feature data based on the second attention weight to obtain weighted initial fused feature data; and fusing the weighted initial fused feature data with the secondary fused feature data to generate a unified multimodal representation of multimodal data.

[0008] Optionally, the acquiring of multimodal feature data includes: acquiring preprocessed multimodal data, wherein the preprocessed multimodal data includes first modal data, second modal data, and third modal data; and based on different feature extraction models, performing feature extraction on the first modal data, the second modal data, and the third modal data, respectively, to acquire the multimodal feature data.

[0009] Optionally, the preprocessing includes: aligning the multimodal data to ensure that the first modal data, the second modal data and the third modal data in the same scene correspond to each other; and standardizing the multimodal data to eliminate differences between the multimodal data.

[0010] Optionally, the first modality data is image data, the second modality data is text data, and the third modality data is audio data.

[0011] Optionally, in the fully connected fusion layer, the initial fused feature data is fused for a second time to form secondary fused feature data, including: splicing each single modal feature data in the initial fused feature data to obtain the spliced ​​initial fused feature data; and in the fully connected fusion layer, mapping the spliced ​​initial fused feature data to a representation space of a different dimension to form secondary fused feature data.

[0012] Optionally, the method obtains a unified multimodal representation of the multimodal data by inputting the multimodal data into a preset unified representation model, wherein the unified representation model includes an encoder, a first attention-based fusion layer, a fully connected fusion layer, a second attention-based fusion layer and an output layer, and the first attention-based fusion layer, the fully connected fusion layer and the second attention-based fusion layer are connected in sequence.

[0013] Optionally, the preset unified representation model is pre-trained in the following manner: obtaining training multimodal data, inputting the multimodal data into the unified representation model, and obtaining a unified multimodal representation of the multimodal data; respectively calculating the consistency loss values ​​between the unimodal data in the multimodal data and the unimodal data in the unified multimodal representation of the multimodal data; weightedly summing all the consistency loss values ​​according to a specific weight coefficient to obtain a joint loss value; updating the model parameter values ​​of the unified representation model according to the joint loss value until the joint loss value converges; and using the unified representation model when the joint loss value converges as the preset unified representation model.

[0014] In a second aspect, an embodiment of the present application provides a multimodal data processing device having the function of implementing the multimodal data processing method based on the unified representation model provided in the first aspect above. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules can be software and / or hardware.

[0015] In one embodiment, the multimodal data processing device includes: an acquisition module, configured to acquire multimodal feature data; a first fusion module, configured to perform initial fusion on the multimodal feature data to acquire initial fusion feature data; a second fusion module, configured to perform secondary fusion on the initial fusion feature data to form secondary fusion feature data; a third fusion module, configured to calculate the similarity between the secondary fusion feature data and the initial fusion feature data; generate a second attention weight based on the similarity; weight the initial fusion feature data based on the second attention weight to acquire weighted initial fusion feature data, and fuse the weighted initial fusion feature data with the secondary fusion feature data; and a representation module, configured to generate a unified multimodal representation of the multimodal data based on the fusion of the weighted initial fusion feature data and the secondary fusion feature data.

[0016] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which includes instructions, which, when executed on a computer, enables the computer to execute the multimodal data processing method based on a unified representation model as described in the first aspect.

[0017] In a fourth aspect, an embodiment of the present application provides a computing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the multimodal data processing method based on the unified representation model described in the first aspect is implemented.

[0018] Compared with the prior art that directly extracts features from multimodal data, the embodiments of the present application first preprocess the multimodal data before extracting features to ensure the consistency and quality of the data. In addition, for the fusion of multimodal data, the prior art uses a method of independently processing each modal data and performing simple fusion at a high level, while simple feature concatenation or fully connected layer fusion cannot fully capture the correlation information between different modal data, thereby affecting the performance of the model.

[0019] Different from this, in the embodiment of the present application, the multimodal feature data is first aligned and initially fused, and on this basis, the multimodal feature data after the initial fusion processing is then multi-layered. Since the embodiment of the present application fuses the multimodal feature data by combining a cross-modal attention mechanism and a multi-layer fusion network, rather than the simple fusion method in the prior art, and on the basis of the initial fused feature data, by fusing low-level features with high-level features, high-resolution and semantically strong graphic features can be obtained, the model improves the efficiency of information fusion as well as the accuracy and comprehensiveness of information fusion, thereby improving the accuracy of the model in graphic and text generation applications.

[0020] In summary, the embodiments of the present application can make full use of the rich information of multimodal data, improve the efficiency of information fusion as well as the accuracy and comprehensiveness of information fusion, thereby improving the accuracy of the model in AI subtitle generation and image and text generation applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] By reading the detailed description of the embodiments of the present application with reference to the accompanying drawings, the objectives, features and advantages of the embodiments of the present application will become easy to understand. Among them:

[0022] By reading the detailed description of the embodiments of the present application with reference to the accompanying drawings, the objectives, features and advantages of the embodiments of the present application will become easy to understand. Among them:

[0023] Figure 1 A schematic diagram of a flow chart of a multimodal data processing method based on a unified representation model according to an embodiment of the present application;

[0024] Figure 2 A schematic diagram of a process of a unified representation model for initially fusing text data and image data in an embodiment of the present application;

[0025] Figure 3 A schematic diagram of a unified representation model for initially fusing text data and audio data in an embodiment of the present application;

[0026] Figure 4 A schematic diagram of a process of a unified representation model for initially fusing text data, audio data, and image data in an embodiment of the present application;

[0027] Figure 5 A schematic diagram of a process of obtaining a unified representation model for secondary fusion feature data in an embodiment of the present application;

[0028] Figure 6 A schematic diagram of a flow chart of a unified representation model for processing text data and image data to obtain graphic and text feature representations in an embodiment of the present application;

[0029] Figure 7 A schematic diagram of the overall structure of a unified representation model for processing text data, image data, and audio data to obtain graphic and text feature representations in an embodiment of the present application;

[0030] Figure 8 A schematic diagram of training a preset unified representation model using task learning in an embodiment of the present application;

[0031] Fig. 9 A schematic diagram of the structure of a data processing device according to an embodiment of the present application; and

[0032] Fig.10 A schematic diagram of the structure of a computing device according to an embodiment of the present application.

[0033] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0035] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0036] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0037] The embodiments of the present application provide a method, apparatus, device and medium for processing multimodal data based on a unified representation model. The method for processing multimodal data based on a unified representation model can be applied to a terminal device, which can be an electronic device such as a tablet computer, a laptop computer, a desktop computer, a personal digital assistant and a wearable device. The terminal device can be a server or a server cluster.

[0038] Some embodiments of the present application are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0039] Please refer to Figure 1 , Figure 1 A flowchart of a multimodal data processing method based on a unified representation model provided in an embodiment of the present application.

[0040] like Figure 1 As shown, the multimodal data processing method based on the unified representation model includes steps S101 to S105.

[0041] Step S101, obtaining multimodal feature data;

[0042] In an embodiment of the present application, multimodal feature data can be obtained based on the original multimodal data, or multimodal feature data transmitted by an external device can be obtained. When obtaining multimodal feature data based on multimodal data, feature data extractors or models of different single modalities can be used to extract feature data of each single modality, and then the feature data of each single modality can be combined into multimodal feature data. For example, image feature data can be extracted by an image feature extractor for image modality data, text feature data can be extracted by a text feature extractor for text modality data, and sound feature data can be extracted by a sound feature extractor for sound modality data, and then the image feature data, text feature data and sound feature data can be combined to obtain multimodal feature data.

[0043] In step S101 of the embodiment of the present application, in order to obtain multimodal feature data, the following steps S1011 to S1012 are included:

[0044] Step S1011: obtaining preprocessed multimodal data, wherein the preprocessed multimodal data includes first modal data, second modal data, and third modal data;

[0045] It should be noted that multimodal data refers to a collection of data from different senses or measurement methods, which usually carry different types of information but together describe the same event or phenomenon. In the field of machine learning and artificial intelligence, the processing of multimodal data can help the system understand complex real-world scenarios more comprehensively. Common multimodal data types include: images and text, for example, a photo and text describing the photo; video and audio, for example, a video and its accompanying sound track; text and audio, for example, an audio file of a speech and the text of the speech, etc.

[0046] In an embodiment of the present application, the multimodal data of the present application includes first modal data, second modal data and third modal data. The first modal data is image data, the second modal data is text data, and the third modal data is audio data. The image data can be photos in formats such as BMP, PNG, etc. The image data can be natural images: such as photos of scenery, people, and objects; it can also be medical images: such as X-rays, MRI (magnetic resonance imaging), CT (computer tomography) images; it can also be satellite images: such as images of the earth's surface taken by remote sensing satellites. Text data can be data in formats such as plain text (.txt) and rich text (.rtf). Text data can also be natural language texts such as news articles, conversation records, social media posts, etc.; it can also be technical documents such as API documents and research papers. Audio data can be data in formats such as MP3, WAV, etc., and audio data can also be voice data such as conversation recordings, voice commands, etc.; it can also be music data such as songs and instrument performances; it can also be environmental sounds such as natural sounds and urban noises.

[0047] Step S1012: Based on different feature extraction models, feature extraction is performed on the first modal data, the second modal data, and the third modal data respectively to obtain multimodal feature data.

[0048] In an embodiment of the present application, when extracting features from each single modal data in the multimodal data, the present application adopts an improved network structure to extract effective features of the first modal data, the second modal data, and the third modal data. The method for extracting features from the single modal data includes the following methods ac:

[0049] Method a

[0050] In an embodiment of the present application, for feature extraction of first modality data, for example, when feature extraction is performed on image data, a residual network (Residual Net, ResNet) convolutional neural network, i.e., a ResNet convolutional neural network, can be used to extract features of the first modality data to obtain first modality feature data, and then the first modality feature data is subjected to dimensionality reduction and normalization processing. Wherein, the first feature data is image data.

[0051] Specifically, the feature extraction of the first modal data can be achieved by using the ResNet convolutional neural network. Specifically, this process can be implemented in a Python program. First, the transforms.compose function is used to combine multiple transformation operations, including resizing, center cropping, conversion to vectors, and normalization of the image data. Then, the torchvision.models.resnet18 function is used to load the pre-trained ResNet18 model, and then the pre-processed image data is input into the model, and the output of the last layer is obtained as the feature. Then, the np.linalg.norm function is used to perform L2 normalization on the features, and finally, principal component analysis is used to perform dimensionality reduction on the features. Through this method, the effective features of the first modal data can be extracted, which enhances the ability to extract features of the first modal data in the single-modal feature extraction stage.

[0052] In an embodiment of the present application, the features of the first modal data can also be extracted by an efficient network (EfficientNet) neural network to obtain the first modal feature data, and the first modal feature data is subjected to dimensionality reduction and normalization processing. The EfficientNet neural network is a convolutional neural network (CNN) model for image recognition, which improves the efficiency and accuracy of the network through a composite scaling method (i.e., adjusting the depth, width, and resolution at the same time).

[0053] Method b

[0054] In an embodiment of the present application, for feature extraction of second modality data, such as feature extraction of text data, a pre-trained Bidirectional Encoder Representations from Transformers (BERT) language model, i.e., a BERT language model, can be used to extract contextual features of the second modality data to obtain second modality feature data, and the second modality feature data can be converted into a vector representation through an embedding layer.

[0055] Specifically, the specific process of using the BERT model to extract contextual features of text is to use a tokenizer to tokenize the text and convert the tokenization results into the input format of the model; then, the processed input data is passed to the BERT model to obtain features, and the embedding representation of the CLS token of the last layer is selected as the feature vector of the text data, and optionally a linear layer is used for dimensionality reduction; finally, the obtained embedding vector is L2 normalized. Through this method, effective features of the second modality data can be extracted, and the ability to extract key features of the second modality data in the single-modality feature extraction stage is enhanced. The CLS token is a key component of the BERT model. It is a learnable embedding vector that is usually placed in the first position of the input sequence, allowing the model to effectively learn and represent the information of the entire input sequence.

[0056] In an embodiment of the present application, the contextual features of the second modality data can also be extracted by a generative pre-trained transformer (GPT) language model, i.e., a GPT language model, to obtain the second modality feature data, and convert the second modality feature data into a vector representation through an embedding layer. The GPT language model is a pre-trained language model based on a transformer architecture, which is mainly used to generate text, can understand natural language and generate coherent and grammatically correct sentences.

[0057] Method c

[0058] In an embodiment of the present application, for feature extraction of third modal data, for example, when feature extraction is performed on audio data, the spectral features of the third modal data can be extracted by using a WaveNet acoustic model to obtain third modal feature data, and the third modal feature data can be reduced in dimension and standardized.

[0059] Specifically, by using the WaveNet acoustic model, the process of extracting the spectral features of audio data includes: first, an audio dataset is required. These audio files should be loaded and converted into a format that the model can handle. A pre-trained WaveNet model is loaded. The spectral features of the audio are extracted using the WaveNet model or its variants. The WaveNet model usually uses a convolutional layer to extract features. If the extracted feature dimension is very high, autoencoders, principal component analysis or other dimensionality reduction techniques can also be used to reduce the dimension of the feature. The features are then standardized to have zero mean and unit variance, which helps the performance of the machine learning algorithm. Through this method, effective features of the third modality data can be extracted, which enhances the ability to extract key features of the third modality data in the single-modality feature extraction stage.

[0060] In the embodiment of the present application, a vector graphic Gaussianization of audio (VGGish) acoustic model, i.e., a VGGish acoustic model, can also be used to extract spectral features of audio data to obtain third modal feature data. VGGish is a deep learning model for sound recognition, but is optimized for sound signals and can extract features from audio clips and identify sound events.

[0061] In the embodiments of the present application, the ability to extract key features of image, audio and text data is improved by using an improved neural network model, an acoustic model and a pre-trained language model.

[0062] In order to make full use of the rich information of multimodal data to ensure the consistency and quality of the data, this application first needs to preprocess the original data of different modalities, where the preprocessing includes: aligning the multimodal data in time and space to ensure that the first modal data, the second modal data, and the third modal data in the same scene can correspond to each other. Alignment processing is conducive to discovering and establishing corresponding relationship information between different modal data, and associating this information. Alignment processing ensures the mutual correlation between different modal data. For example, in video processing, sound and picture require strict time synchronization; in scenes where text and images are combined, it is necessary to find the entities in the text description and the corresponding parts in the image.

[0063] In the embodiments of the present application, data of different modalities may be aligned in time and space to ensure that image data, text data, and audio data in the same scene can correspond to each other. For example, timestamps may be used to align audio and video data, and object detection algorithms may be used to align object descriptions in images and text.

[0064] Specifically, temporal data alignment is to bind different modal data collected at a time with a timestamp. In this way, a time point corresponds to data of different modalities. For example, there is a video dataset that contains video frames and audio clips. The video frames and audio clips need to be temporally aligned for use in subsequent multimodal tasks, where the video frame is a series of image frames at 25 frames per second; the audio clip is a piece of audio data with a sampling rate of 44.1kHz. First, load the video frame and audio data. Resample the audio data to the same sampling rate as the video frame. Assign a timestamp to each video frame and audio clip. Ensure that data of different modalities are temporally aligned through timestamps. Through these steps, the consistency of multimodal data can be ensured. In multimodal data processing, temporal alignment ensures that the information of data of different modalities at the same time point is synchronized. This is crucial for subsequent data analysis, feature extraction, and model training, because data of different modalities often need to be combined to provide complete contextual information.

[0065] When aligning image data and text data using object detection algorithms, key feature points such as corners, edges, etc. are extracted from the image. For text data, words, phrases, or sentences can be extracted as features. The image features are matched with the text features. Based on the matched feature points, a transformation model (such as a homography matrix) is established to map the image data to the reference frame of the text description. This may involve transformations such as rotation, scaling, and translation of the image. The transformation model is optimized to minimize the difference between the image and the text. Finally, the detected objects are aligned with the objects in the text description to ensure the consistency and quality of the multimodal data.

[0066] In addition to preprocessing the multimodal data by aligning the multimodal data in time and space, the multimodal data may also be standardized to eliminate the differences between the multimodal data.

[0067] In an embodiment of the present application, when standardizing multimodal data, the differences between different modalities can be eliminated by unifying image data into the same resolution and color space and converting text data into the same language representation format.

[0068] Specifically, when the image data is unified to the same resolution and color space, this goal is achieved by using Python and related libraries such as OpenCV and PIL. The cv2.resize function in the OpenCV library is used to adjust the image size, and the cv2.cvtColor function is used to convert the color space. By default, OpenCV uses the BGR color space, while most image processing libraries (such as PIL) use the RGB color space. Finally, the functions of adjusting the resolution and converting the color space are combined in one function. Among them, the appropriate resolution is selected according to the needs. For example, if you process high-resolution images, you may need to use a larger target resolution to preserve details. By unifying the image data to the same resolution and color space, this helps ensure the consistency of the data, thereby improving the quality of subsequent processing or analysis.

[0069] When converting text data to the same language representation format, you can use Python and related libraries (such as the transformers library) to achieve this goal. First, clean the text data, including removing irrelevant characters, standardizing punctuation, converting uppercase and lowercase, and so on. Then split the text data into words or tokens, remove stop words in the text, convert words to their basic form, and finally use a pre-trained word embedding model to convert the text data to a unified representation format. Converting text data to the same language representation format helps ensure the consistency of the data, thereby improving the quality of subsequent processing or analysis.

[0070] This application ensures high-quality matching and consistency of multimodal data by adopting advanced data alignment technology and modality standardization.

[0071] Step S102: In the first attention-based fusion layer, the multimodal feature data is converted into a query vector, a key vector and a value vector respectively, the first attention weight between the query vector and the key vector is calculated, the value vector is weighted based on the first attention weight to obtain the weighted value vector, the weighted value vector is initially fused with the query vector to obtain the initial fused feature data.

[0072] In an embodiment of the present application, in order to obtain initial fused feature data, the multimodal feature data is first converted into a query (Query, Q) vector, a key (Key, K) vector and a value (Value, V) vector based on a cross-modal attention mechanism; the similarity between the query vector and the key vector is calculated, and a first attention weight is calculated based on the similarity, and then the value vector is weighted based on the first attention weight to obtain a weighted value vector, and finally the weighted value vector is initially fused with the query vector to obtain initial fused feature data, which is conducive to utilizing the rich information of multimodal data and improving the efficiency of information fusion.

[0073] Among them, the method of obtaining Q, K and V is usually to project the input data through linear transformation. The input data is usually an embedded vector sequence, such as word embedding in natural language processing (NLP), and image block embedding in computer vision. In a standard multi-head attention layer, the input data will be transformed through three different linear transformations (weight matrices) to obtain Q, K, and V. It should be noted that in the cross-modal attention mechanism, Q, K and V are extended to process data of different modalities, such as Q is used to represent or process text data, K and V are used to represent or process image data, etc.

[0074] In some embodiments, according to different application requirements, a cross-modal attention mechanism is used to initially fuse any two feature data or all feature data in the multimodal feature data. For example, in a graphic and text generation application, the image and text feature data are initially fused, so that the model can focus on the key areas in the image and correspond to the keywords in the text, thereby ensuring that the key features of the multimodal feature data are effectively integrated.

[0075] For example, in AI subtitle automatic generation applications, by initially fusing audio and text features, it is possible to more accurately capture key information in the audio and generate subtitles synchronized with the audio content. For another example, in multimodal sentiment analysis, by initially fusing text features, image features, and audio features, it is possible to better understand the multimodal context of emotional expression.

[0076] In some embodiments, see the attached Figure 2 , Figure 2 This is a flow chart of a unified representation model for initially fusing text data and image data in one embodiment of the present application. Figure 2 In the application of image and text generation, for example, a deep learning model can be designed. The deep learning model first uses the BERT model to extract features from text data and the ResNet model to extract features from image data, respectively obtaining text feature data and image feature data. Then, the image feature data and text feature data are initially fused based on the cross-modal attention mechanism to obtain initial fused feature data.

[0077] Among them, in unimodal data, Q can be regarded as a "question" signal used to find the most relevant contextual information. K contains the representation of all potential relevant information, which is used to compare with the query vector to determine which information the query vector should focus on. V contains the actual content information, that is, K contains the specific information of each word or sentence part, which will be weighted after calculating the attention weight.

[0078] In multimodal data, the distribution of Q, K, and V is not fixed, but needs to be determined based on the actual application scenario and the characteristics of multimodal data. In a trained model, one modal data (such as audio data) is usually used as input (i.e., Q), and the model is used to predict the output of another modal data (such as text data). At this time, K and V come from the text modality feature space in the trained model, and the modality from which Q, K, and V come actually corresponds to the actual task. For example, when generating subtitles using sound, audio data is used as input, and the text content is predicted based on the model. Therefore, audio data is Q, and text data is K and V.

[0079] What the model training process actually needs to do is to find correlations in different modal data to infer what kind of representation will appear in another modal data when one modal data presents a representation. In this process, the modal data used as input in the actual task can be used as Q. During the model training process, in order to utilize complementary information, the model may also use text data as Q to perform reverse mapping from text to audio. This method can enhance the model's ability to utilize complementary information.

[0080] What the cross-modal attention mechanism does is, for example, which features in the first modal features should be paid more attention to when inferring the second modal data representation, and naturally have higher weights in the weight matrix, so there will be a correlation calculation. The modality of V is actually the feature of the temporary second modal data (temporary output modality) during training. It is multiplied by the calculation results of Q and K to obtain the weighted second modal feature, which can be understood as some of the features that should be paid more attention to are magnified.

[0081] The cross-modal attention mechanism can learn the complex relationship between different modal data, capture the complementary information between them, and achieve efficient cross-modal generation or conversion on a limited data set by flexibly using Q, K, and V and alternating the positions of different modal data in Q, K, and V. This mechanism not only improves the performance of the model, but also enhances the model's ability to utilize complementary information.

[0082] For example, text data usually contains clear semantic information, which can be used as Q to ask "questions" or guide the model to find relevant information in other modal data. Image data may contain a large number of visual features, which can be used as K to help the model identify which visual information is related to text information. Audio data may contain non-verbal clues such as emotions and tone, which can be used as V because they provide complementary information, which is crucial to understanding the overall situation in some cases. In multimodal data, the appropriate allocation of Q, K, and V can help the model better understand and integrate information from different modalities. In some applications, text data may need to be used as the dominant source of information. In this case, text data can be used as the query vector Q, while other modal data can be used as the key vector K and the value vector V. If the two modal data are equally important, one modality can be used as Q and the other as the key vector K, and V may come from any modality or a combination of the two. Among them, Q, K, and V all come from the linear transformation of the input data.

[0083] In an embodiment of the present application, taking the generation of images through text in a graphic and text generation application as an example, Q1 in the model is derived from text feature data and represents the representation of text feature data, K1 is derived from image feature data and represents the representation of image feature data; V1 also comes from image feature data, where K1 is used to calculate the first attention weight of the text data with Q1, and V1 contains specific information of the image data.

[0084] In an embodiment of the present application, for example, there are two data sets of modal feature data, one is an image feature data set, and the other is a text feature data set. These data sets are usually derived from pairs of images and texts, and each pair represents a specific scene or object. For example, a data set may be a photo of a kitten, with a description: "A white cat on a mat". Among them, the data in the image feature data set may be feature data whose features have been extracted by a ResNet convolutional neural network model, and the data in the text feature data set may be feature data whose features have been extracted by a BERT language model or a GPT language model, and then the attention mechanism module in the deep neural network model is used to obtain data that combines text and image information.

[0085] In the cross-modal attention mechanism, the similarity matrix calculates the similarity between Q and K through a dot product operation, which reflects the correlation between different features. Then, the similarity matrix is ​​normalized to obtain the first attention weights, which are used to weight V to generate new feature representations. In this process, the model can learn which input modal feature data is more important for the predicted output modal feature representation, thereby achieving "amplification" of these features.

[0086] Specifically, see Figure 2, usually the image feature data and text feature data are linearly transformed to generate Q1, K1 and V1. Specifically, the text feature data, image feature data and audio feature data in the encoder are multiplied by three matrices to obtain Q1, K1 and V1 vectors, which are obtained through training. Then, the similarity is calculated by calculating the dot product between Q1 and K1 to obtain the similarity matrix, and then the normalized exponential (Softmax) function is used to normalize the similarity matrix to obtain the first attention weight. The first attention weight is then used to perform weighted averaging on V1 (image feature) to highlight the image area or feature most relevant to the text description. The weighted V1 (image feature) and Q1 (text feature) are then initially fused by a linear weighted fusion method to obtain initial fused feature data, which represents data combining text and image information. The initial fused feature data is used as input data for multi-level fusion. Through this mechanism, the key areas in the image can be focused on and correspond to the keywords in the text, thereby ensuring that the key features of the multimodal feature data are effectively integrated.

[0087] In the examples of this application, see Figure 3 , Figure 3 A flowchart of a unified representation model for initial fusion of text data and audio data in an embodiment of the present application. Taking the AI ​​subtitle automatic generation application as an example, firstly, the BERT model is used to extract features for text data, and the WaveNet model is used to extract features for audio data, and text feature data and audio feature data are obtained respectively. Q2 comes from the text feature data, representing the query vector of the subtitle, K2 and V2 come from the audio feature data, and then the dot product or other similarity measurement methods are used to calculate the similarity between Q2 and K2 to obtain the similarity matrix, and then the similarity matrix is ​​normalized by the Softmax function to obtain the first attention weight, which is then used to weighted average V2 (audio feature), and then the weighted V2 (audio feature) and Q2 (text feature) are initially fused by the linear weighted fusion method to obtain the initial fused feature data, which represents the data combining text and audio information, and the initial fused data is used as the input data of the fully connected fusion layer. Through this mechanism, the audio information can be effectively combined with the text information, and the key information in the audio can be captured more accurately to generate accurate subtitles.

[0088] In the examples of this application, see Figure 4 , Figure 4 This is a flow chart of a unified representation model for initially fusing text data, audio data, and image data in one embodiment of the present application. Figure 4In this paper, suppose you are building a sentiment analysis system for analyzing social media posts (containing text, images, and accompanying video or audio). The system needs to identify the emotional tendency in the post, which may require considering the text description, the emotional expression in the image, and the emotional clues in the audio (such as the user's voice or background music).

[0089] First, use the pre-trained BERT to convert the text into a feature vector; then use ResNet to extract the key features of the image data and possibly encode these features into a series of vectors. Then use an audio processing model (such as WaveNet) to extract the spectral and temporal features of the audio, and also encode them into a vector sequence. Design a model that includes a Transformer layer or a similar attention layer with a QKV mechanism to handle cross-modal interactions. In this layer, Q3 is generated from the text feature vector to guide the model to find relevant information in other modal data. K3 and V3 are generated from image and audio feature vectors, respectively, so that the information in the text can "query" the relevant parts in the image and audio. The similarity is calculated by calculating the dot product between Q3 and K3 to obtain the similarity matrix, and then the Softmax function is used to normalize the similarity matrix to obtain the first attention weight. The first attention weight is then used to perform weighted averaging on V3, and then the weighted V3 and Q3 are fused by a linear weighted fusion method to finally obtain the initial fused feature data, which combines the data of text, image and audio information. The initial fused feature data is used as the input data for multi-level fusion so that the multimodal context of emotional expression can be better understood.

[0090] Step S103: In the fully connected fusion layer, the initial fused feature data is fused for a second time to form secondary fused feature data.

[0091] In the examples of this application, see Figure 5 , Figure 5 A flowchart of a unified representation model for obtaining secondary fusion feature data in one embodiment of the present application. In order to obtain secondary fusion feature data, each single-modal feature data in the initial fusion feature data is first spliced ​​using a direct splicing method to obtain the spliced ​​initial fusion feature data; then a fully connected fusion layer is used to map the spliced ​​initial fusion feature data to a representation space of different dimensions, that is, a secondary fusion process is performed to form secondary fusion feature data, which is conducive to further utilizing the richer information of multimodal data and further improving the efficiency of information fusion.

[0092] In an embodiment of the present application, taking the initial fusion feature data as the fusion of image feature data and text feature data as an example, the image feature data and text feature data in the initial fusion feature data are first spliced, that is, the image feature data and the text feature data are spliced ​​together; then a fully connected layer is used to map the spliced ​​features to a higher-dimensional representation space, for example, 128 dimensions before and 256 dimensions after mapping, so that the feature vector may contain more information, which is beneficial to the accuracy and comprehensiveness of information fusion.

[0093] Step S104: In the second attention-based fusion layer, the similarity between the secondary fusion feature data and the initial fusion feature data is calculated, a second attention weight is generated based on the similarity, the initial fusion feature data is weighted based on the second attention weight, and the weighted initial fusion feature data is obtained.

[0094] In an embodiment of the present application, the attention-based fusion layer is a branching layer in a deep learning model. In an embodiment of the present application, multi-level fusion is performed on the basis of the initial fusion feature data in step S102. This strategy can make full use of the rich information of multimodal data, improve the efficiency of information fusion, and improve the accuracy and comprehensiveness of information fusion, thereby improving the accuracy of the model in AI subtitle generation and image and text generation applications.

[0095] Step S105: Fusing the weighted initial fused feature data with the secondary fused feature data to generate a unified multimodal representation of the multimodal data.

[0096] In the examples of this application, see Figure 6 , Figure 6 The present invention is a flowchart of a unified representation model for processing text data and image data to obtain text-image feature representation in an embodiment of the present invention. Taking the text-image generation application as an example, on the basis of obtaining the initial fused feature data through the cross-modal attention mechanism, the initial fused feature data containing text and image information is secondary fused using a fully connected fusion layer to form secondary fused feature data, and on the basis of the secondary fused feature data, the attention-based fusion layer is used to perform a third fusion process on the secondary fused feature data.

[0097] Specifically, the text features in the secondary fusion feature data are first converted into Q4, K4 and V4 as image features in the initial fusion feature data, and the second attention weight is generated by calculating the similarity between Q4 and K4 by using dot product or multiplication. The second attention weight is weighted on the initial fusion feature data to generate weighted initial fusion feature data. Finally, the weighted initial fusion feature data and the secondary fusion feature data are fused for the third time to generate a unified multimodal representation of multimodal data. The third fusion can be achieved by weighted averaging or by another fully connected layer, thereby generating a more refined representation of image and text features. Based on the initial fusion feature data, by fusing low-level features with high-level features, high-resolution and semantically strong image and text features can be obtained. The model improves the efficiency of information fusion as well as the accuracy and comprehensiveness of information fusion, thereby improving the accuracy of the model in image and text generation applications.

[0098] In the embodiment of the present application, by adopting a combination of a cross-modal attention mechanism and a multi-level fusion network, that is, a hybrid fusion method, it can make full use of the rich information of multimodal data and improve the efficiency of information fusion. The fully connected layer can capture the complex interactions between different modal features and extract richer and more useful feature representations through nonlinear transformations.

[0099] In the embodiments of the present application, due to insufficient information fusion and high demand for computing resources, the performance of existing models is often unsatisfactory, especially in terms of generalization ability and the ability to adapt to different tasks. Existing models are usually optimized for specific tasks, and therefore lack generalization ability in a multi-task environment, and it is difficult to maintain excellent performance in a variety of application scenarios. The embodiments of the present application calculate the consistency loss value between the unimodal data in the multimodal data and the unimodal data in the unified multimodal representation of the multimodal data, respectively, and then weighted sum all the consistency loss values ​​according to a specific weight coefficient to obtain a joint loss value, and then update the model parameter values ​​of the unified representation model according to the joint loss value until the joint loss value converges, and finally train the unified representation model when the joint loss value converges as the preset unified representation model, thereby improving the model's representation ability for different modal data and realizing an efficient, flexible and adaptable multimodal learning solution. The method obtains a unified multimodal representation of the multimodal data by inputting the multimodal data into a preset unified representation model, wherein the unified representation model includes an encoder, a first attention-based fusion layer, a fully connected fusion layer, a second attention-based fusion layer, and an output layer.

[0100] In the examples of this application, see Figure 7 , Figure 7This is a schematic diagram of the overall structure of a unified representation model for processing text data, image data, and audio data to obtain graphic and text feature representations in one embodiment of the present application. Figure 7 In the model structure, the encoder is used to extract features from image data, text data, and audio data. The encoder includes an image encoder, a text encoder, and an audio encoder, wherein the image encoder can be based on a pre-trained convolutional neural network (CNN), such as ResNet50; the text encoder can be a BERT language model; the audio encoder can be based on a WaveNet acoustic model, or a hybrid architecture based on a convolutional neural network (CNN) and a recurrent neural network (RNN).

[0101] In the examples of this application, see Figure 7 The first attention-based fusion layer is a layer based on the attention mechanism, which is embedded as a module in the multi-level fusion network of the unified representation model. It is located before the second attention-based fusion layer of the fully connected fusion layer to perform initial fusion on the multimodal feature data, thereby ensuring that the key features of the multimodal feature data are effectively integrated. The initial fusion feature data is then secondary fused in the fully connected fusion layer to form secondary fusion feature data, and then in the second attention-based fusion layer, the similarity between the secondary fusion feature data and the initial fusion feature data is calculated, and the second attention weight is generated based on the similarity. The initial fusion feature data is weighted based on the second attention weight to obtain the weighted initial fusion feature data, and the weighted initial fusion feature data is fused with the secondary fusion feature data to generate a unified multimodal representation of the multimodal data.

[0102] In the examples of this application, see Figure 8 , Figure 8 This is a schematic diagram of training a preset unified representation model using task learning in one embodiment of the present application. Figure 8 In the embodiment, the encoder is used to extract features from image data, text data and audio data, and the multi-level fusion layer is used to perform multi-level fusion of multi-modal feature data. The output layer is used to output a unified multi-modal representation of the generated multi-modal data. The trained multi-modal data includes image data, text data and audio data. The image data, text data and audio data are input into a unified representation model to obtain a unified multi-modal representation of the multi-modal data. The sequence loss function is used to calculate the consistency loss value between the text data in the multi-modal data and the text data in the unified multi-modal representation of the multi-modal data. The cross-entropy loss function is used to calculate the consistency loss value between the image data in the multi-modal data and the image data in the unified multi-modal representation of the multi-modal data. The cross-entropy loss function is used to calculate the consistency loss value between the audio data in the multi-modal data and the audio data in the unified multi-modal representation of the multi-modal data.

[0103] Afterwards, all loss values ​​are weighted and summed according to a specific weight coefficient, and the weight is adjusted through preliminary experiments to observe the model performance under different weight configurations, where the specific weight coefficient can be 0.3, 0.3 and 0.4, and the specific weight coefficient can be adjusted through experiments. Obtain a joint loss value; update the model parameter value of the unified representation model according to the joint loss value until the joint loss value converges; and use the unified representation model when the joint loss value converges as the preset unified representation model.

[0104] The unified representation model can process and integrate features from different modalities, and apply the trained unified representation model to actual multimodal tasks, such as image-text generation (image-text), AI subtitle generation (speech-text), etc. For example, if you are building an AI subtitle generation system, the system needs to convert audio signals into text. First, you need to use the WaveNet acoustic model to preprocess the audio data, including feature extraction, dimensionality reduction and standardization of the audio data, and adjust the volume level of the audio to make it suitable for the speech recognition system. Then use the trained unified representation model to recognize the speech in the audio and convert it into text.

[0105] The unified representation model can be trained through multi-task learning to improve the performance and adaptability of the model in various tasks and applications, enhance the generalization ability of the model, reduce computing resource consumption, and increase the training and reasoning speed of the model. The performance and adaptability of the model in various tasks and applications can also be improved through self-supervised learning, and the generalization ability of the model can be enhanced.

[0106] This exemplary embodiment proposes a multimodal data processing device, such as Fig. 9 The structural schematic diagram of the data processing device shown provides a method that can be applied to a server in a scenario where multimodal data processing is required, which includes obtaining multimodal feature data, using a cross-modal attention mechanism to initially fuse the multimodal feature data, obtaining initial fused feature data, and then using a multi-level fusion network to perform multi-level fusion on the initial fused feature data to generate a unified representation model; and generating a unified modal representation of the multimodal data based on the unified representation model.

[0107] The data processing device in the embodiment of the present application can implement the above Figure 1 The steps of the multimodal data processing method based on the unified representation model executed in the corresponding embodiment. The functions implemented by the data processing device can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules can be software and / or hardware.

[0108] In the examples of this application, see Fig. 9 The multimodal data processing device 1 comprises: an acquisition module 11, a first fusion module 12, a second fusion module 13, a third fusion module 14 and a representation module 15, wherein:

[0109] The acquisition module 11 is configured to acquire multimodal feature data.

[0110] In an embodiment of the present application, multimodal data includes first modal data, second modal data, and third modal data. The first modal data is image data, the second modal data is text data, and the third modal data is audio data. Data from different modalities can be aligned in time and space to ensure that the image, text, and audio data in the same scene can correspond. For example, timestamps can be used to align audio and video data, and object detection algorithms can be used to align object descriptions in images and texts. Compared with the prior art of directly extracting features from multimodal data, some embodiments of the present application first pre-process the multimodal data before performing feature extraction to ensure data consistency and quality.

[0111] The first fusion module 12 is configured to perform initial fusion on the multimodal feature data to obtain initial fused feature data.

[0112] In an embodiment of the present application, the first fusion module 12 can use a cross-modal attention mechanism to initially fuse the multimodal feature data, which includes: converting the multimodal feature data into a query vector, a key vector, and a value vector respectively; calculating the first attention weight between the query vector and the key vector; weighting the value vector based on the first attention weight to obtain the weighted value vector, and initially fusing the weighted value vector with the query vector to obtain initial fused feature data. By initially fusing the multimodal feature data, it is ensured that the key features of the multimodal feature data are effectively integrated, which is conducive to the subsequent multi-level fusion to provide rich information on the key features of the multimodal feature data.

[0113] The second fusion module 13 is configured to perform secondary fusion on the initial fused feature data by using a fully connected fusion layer to form secondary fused feature data.

[0114] In an embodiment of the present application, in order to obtain secondary fusion feature data, each single-modal feature data in the initial fusion feature data is first spliced ​​using a direct splicing method to obtain the spliced ​​initial fusion feature data; then a fully connected fusion layer is used to map the spliced ​​initial fusion feature data to a representation space of a different dimension, that is, a secondary fusion process is performed to form secondary fusion feature data, which is conducive to further utilizing the richer information of multimodal data and further improving the information fusion efficiency.

[0115] The third fusion module 14 is configured to use the attention-based fusion layer to calculate the similarity between the secondary fusion feature data and the initial fusion feature data, generate a second attention weight based on the similarity, weight the initial fusion feature data based on the second attention weight, obtain the weighted initial fusion feature data, and fuse the weighted initial fusion feature data with the secondary fusion feature data.

[0116] In the embodiment of the present application, on the basis of obtaining the initial fused feature data through the cross-modal attention mechanism, the fully connected fusion layer is used to perform secondary fusion on the initial fused feature data containing text and image information to form secondary fused feature data. On the basis of the secondary fused feature data, the third fusion module 14 uses the attention-based fusion layer to perform a third fusion process on the secondary fused feature data. On the basis of the initial fused feature data, by fusing low-level features with high-level features, high-resolution and semantically strong graphic features can be obtained. The model improves the efficiency of information fusion as well as the accuracy and comprehensiveness of information fusion, thereby improving the accuracy of the model in graphic and text generation applications.

[0117] The generating module 15 is configured to fuse the weighted initial fused feature data and the secondary fused feature data to generate a unified multimodal representation of the multimodal data.

[0118] The embodiment of the present application adopts an improved network structure to extract effective features of each modality, enhances the ability of multimodal feature extraction in the single-modal feature extraction stage, and pre-processes the multimodal data before feature extraction to ensure the consistency and quality of the data. The first fusion module aligns and initially fuses the multimodal feature data by adopting a cross-modal attention mechanism, and initially fuses the multimodal feature data to ensure that the key features of the multimodal feature data are effectively integrated, which is conducive to the subsequent multi-level fusion to provide rich information of the key features of the multimodal feature data. The initial fusion feature data is subjected to feature splicing and secondary fusion processing of the fully connected layer by the second fusion module. By adopting a combination of a cross-modal attention mechanism and a deep fusion network, the rich information of the multimodal data is fully utilized, the information fusion efficiency is improved, and on the basis of the initial fusion feature data, by fusing low-level features and high-level features, high-resolution and semantically strong graphic features can be obtained. The model improves the information fusion efficiency and the accuracy and comprehensiveness of information fusion, thereby improving the accuracy of the model in graphic and text generation applications. A multi-task learning approach is used to train the unified representation model to improve the model's ability to represent data of different modalities and to achieve an efficient, flexible and adaptable multimodal learning solution.

[0119] To achieve the above purpose, the present application also provides a computer device 2, see Fig.10 , Fig.10 A schematic diagram of a structure of a computing device according to an embodiment of the present application is shown in FIG. Fig.10 In the embodiment, the computer device 2 includes multiple computer devices 2, and the components of the multimodal data processing device 1 can be dispersed in different computer devices 2. The computer device 2 can be a smart phone, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server or cabinet server (including independent server, or server cluster composed of multiple servers) that executes the program. The computer device 2 of this embodiment includes at least but is not limited to: a memory 21, a processor 23, a network interface 22 and a multimodal data processing device 1 (refer to Fig.10 ). It should be pointed out that Fig.10 Only a computer device 2 having components is shown, but it should be understood that implementing all of the components shown is not a requirement and more or fewer components may alternatively be implemented.

[0120] In this embodiment, the memory 21 includes at least one type of computer-readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 21 can be an internal storage unit of the computer device 2, such as a hard disk or memory of the computer device 2. In other embodiments, the memory 21 can also be an external storage device of the computer device 2, such as a plug-in hard disk equipped on the computer device 2, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 21 can also include both the internal storage unit of the computer device 2 and its external storage device. In this embodiment, the memory 21 is generally used to store the operating system and various application software installed on the computer device 2, such as the program code of the multimodal data processing method of the present application. In addition, the memory 21 can also be used to temporarily store various types of data that have been output or are to be output.

[0121] The processor 23 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 23 is generally used to control the overall operation of the computer device 2, such as executing control and processing related to data interaction or communication with the computer device 2. In this embodiment, the processor 23 is used to run the program code stored in the memory 21 or process data, such as running the multimodal data processing device 1.

[0122] The network interface 22 may include a wireless network interface or a wired network interface, and the network interface 22 is generally used to establish a communication connection between the computer device 2 and other computer devices 2. For example, the network interface 22 is used to connect the computer device 2 to an external terminal through a network, and to establish a data transmission channel and a communication connection between the computer device 2 and the external terminal. The network may be a wireless or wired network such as an Intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth, or Wi-Fi.

[0123] It should be pointed out that Fig.10 Only computer device 2 is shown with components 21 - 23 , but it should be understood that implementation of all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0124] In this embodiment, the multimodal data processing device 1 stored in the memory 21 can also be divided into one or more program modules, and the one or more program modules are stored in the memory 21 and executed by one or more processors (processor 23 in this embodiment) to complete this application.

[0125] To achieve the above purpose, the present application also provides a computer-readable storage medium, which includes multiple storage media, such as flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, App application store, etc., on which a computer program is stored, and the program implements the corresponding function when executed by the processor 23. The computer-readable storage medium of this embodiment is used to store the multimodal data processing device 1, and when executed by the processor 23, the multimodal data processing method of embodiment 1 is implemented.

[0126] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0127] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0128] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0129] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0130] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0131] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0132] The computer program product includes one or more computer instructions. When the computer program is loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0133] The technical solutions provided in the embodiments of the present application are introduced in detail above. The principles and implementation methods of the embodiments of the present application are explained by using specific examples in the embodiments of the present application. The description of the above embodiments is only used to help understand the methods and core ideas of the embodiments of the present application. At the same time, for those skilled in the art, according to the ideas of the embodiments of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the embodiments of the present application.

Claims

1. A multimodal data processing method based on a unified representation model, the method comprising: Acquire multimodal feature data; In a first attention-based fusion layer, the multimodal feature data is converted into a query vector, a key vector, and a value vector respectively; Calculating a first attention weight between the query vector and the key vector; Performing weighted processing on the value vector based on the first attention weight to obtain a weighted value vector; Initially fusing the weighted value vector with the query vector to obtain initial fused feature data; In the fully connected fusion layer, the initial fusion feature data is secondary fused to form secondary fusion feature data; At a second attention-based fusion layer, calculating the similarity between the secondary fused feature data and the initial fused feature data; generating a second attention weight based on the similarity; Weighting the initial fused feature data based on the second attention weight to obtain weighted initial fused feature data; as well as The weighted initial fused feature data is fused with the secondary fused feature data to generate a unified multimodal representation of the multimodal data.

2. The method of claim 1, wherein: The acquiring of multimodal feature data comprises: Acquiring preprocessed multimodal data, wherein the preprocessed multimodal data includes first modal data, second modal data, and third modal data; and Based on different feature extraction models, feature extraction is performed on the first modal data, the second modal data, and the third modal data respectively to obtain the multimodal feature data.

3. The method of claim 2, wherein: The pre-processing comprises: performing alignment processing on the multimodal data to ensure that the first modal data, the second modal data, and the third modal data in the same scene correspond to each other; and The multimodal data are standardized to eliminate differences between the multimodal data.

4. The method of claim 2, wherein: The first modality data is image data, the second modality data is text data, and the third modality data is audio data.

5. The method of claim 1, wherein: In the fully connected fusion layer, the initial fused feature data is secondary fused to form secondary fused feature data, including: splicing each single-modal feature data in the initial fused feature data to obtain spliced ​​initial fused feature data; and In the fully connected fusion layer, the concatenated initial fused feature data is mapped to a representation space of different dimensions to form secondary fused feature data.

6. The method of claim 1, wherein: The method obtains a unified multimodal representation of the multimodal data by inputting the multimodal data into a preset unified representation model, wherein the unified representation model includes an encoder, a first attention-based fusion layer, a fully connected fusion layer, a second attention-based fusion layer and an output layer, and the first attention-based fusion layer, the fully connected fusion layer and the second attention-based fusion layer are connected in sequence.

7. The method of claim 1, wherein: The preset unified representation model is pre-trained in the following way: Acquire multimodal data for training, input the multimodal data into the unified representation model, and obtain a unified multimodal representation of the multimodal data; respectively calculating consistency loss values ​​between unimodal data in the multimodal data and unimodal data in a unified multimodal representation of the multimodal data; All the consistency loss values ​​are weighted and summed according to a specific weight coefficient to obtain a joint loss value; updating the model parameter values ​​of the unified representation model according to the joint loss value until the joint loss value converges; and The unified representation model when the joint loss value converges is used as the preset unified representation model.

8. A multimodal data processing device, comprising: An acquisition module, configured to acquire multimodal feature data; A first fusion module is configured to perform initial fusion on the multimodal feature data to obtain initial fused feature data; A second fusion module is configured to perform secondary fusion on the initial fusion feature data to form secondary fusion feature data; a third fusion module, configured to calculate the similarity between the secondary fusion feature data and the initial fusion feature data, generate a second attention weight based on the similarity, weight the initial fusion feature data based on the second attention weight, obtain the weighted initial fusion feature data, and fuse the weighted initial fusion feature data with the secondary fusion feature data; as well as The representation module is configured to generate a unified multimodal representation of the multimodal data based on the fusion of the weighted initial fused feature data and the secondary fused feature data.

9. A computing device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium comprising instructions, which, when executed on a computer, causes the computer to perform the method according to any one of claims 1 to 7.

Images