Multimodal joint learning methods, devices, equipment, readable storage media, and products

By directly acquiring the byte sequence of single-modal data for multimodal learning, the complex preprocessing of token conversion is avoided, resource consumption is reduced, and the flexibility and adaptability of the model are improved. This solves the problems of high resource consumption and insufficient flexibility in existing multimodal joint learning methods.

CN118261224BActive Publication Date: 2026-11-17中移信息技术有限公司 +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410350410.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-25
Publication Date
2026-11-17
Estimated Expiration
2044-03-25

AI Technical Summary

Technical Problem

Existing multimodal joint learning methods require complex preprocessing steps, such as vocabulary building and token allocation, which leads to high computer system resource consumption and a lack of flexibility and adaptability in dynamic modality change scenarios.

Method used

By directly acquiring the byte sequence of single-modal data and training it through a fusion network and a task network, the complex preprocessing of converting data into tokens is avoided. The multimodal learning model is trained directly based on the byte sequence, and the network parameters are adjusted until the iteration termination condition is met.

Benefits of technology

It reduces the computer system resource consumption required for multimodal joint learning, improves the flexibility and adaptability of the model, especially when the number of modalities changes, it can dynamically adjust attention weights and reduce the need to modify the model structure.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118261224B_ABST
    Figure CN118261224B_ABST
Patent Text Reader

Abstract

This application discloses a multimodal joint learning method, apparatus, device, readable storage medium, and product, relating to the field of computer technology. The method includes: acquiring training data and a multimodal learning model to be trained, wherein the multimodal learning model includes a fusion network and a task network; acquiring byte sequences corresponding to each type of single-modal data in the training data, inputting each byte sequence into the multimodal learning model, determining fusion embedding vectors corresponding to all byte sequences through the fusion network; determining the processing result corresponding to the fusion embedding vectors through the task network, adjusting the network parameters of the fusion network and the task network based on the processing result, and returning to execute the step of inputting each byte sequence into the multimodal learning model until a preset iteration termination condition is met. This application reduces the computer system resource consumption required to implement multimodal joint learning.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a multimodal joint learning method, apparatus, device, readable storage medium, and product. Background Technology

[0002] In recent years, with the rapid development of internet technology, data has become increasingly diversified. The same object can be described in different ways or from different perspectives, and each perspective or method of this data can be considered a modality. Furthermore, there are certain correlations between different modalities. Therefore, the processing of cross-modal data is particularly important.

[0003] Currently, with the development of internet technology, joint learning techniques for multimodal data have emerged. These techniques mainly involve inputting data from different modalities, utilizing the correlations between these modalities, and training the data through neural networks.

[0004] Multimodal joint learning typically focuses on pre-establishing a vocabulary that stores the correspondence between different characters and tokens (identifiers). By querying the vocabulary, data from different modalities are converted into tokens, and these tokens are then used to train a multimodal deep learning model. For example, unimodal and multimodal data can be input into a decoupled attention transformation (Transformer) network model (a neural network model based on self-attention mechanism) to generate semantic element token semantic representation features and cross-modal semantic representation features, which are then used to train the decoupled attention transformation network model. This allows the decoupled attention transformation network model to effectively utilize multiple different modalities of data to enhance each other, obtaining more robust and powerful general semantic representation capabilities from multimodal data. It can also use a unified network structure model to process data from different modalities and simultaneously perform the tasks of understanding and generating data from different modalities. This multimodal joint learning learning and training method usually requires complex preprocessing steps, resulting in high computer system resource consumption for implementing multimodal joint learning. For example, building and storing the vocabulary requires storage resources, and matching the vocabulary and allocating tokens requires processor resources.

[0005] Therefore, how to reduce the computer system resource consumption required to realize multimodal joint learning is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] The main objective of this application is to provide a multimodal joint learning method, apparatus, device, readable storage medium, and product, aiming to solve the technical problem of how to reduce the computer system resource consumption required to implement multimodal joint learning.

[0007] To achieve the above objectives, this application provides a multimodal joint learning method, which includes the following steps:

[0008] Acquire training data and a multimodal learning model to be trained, wherein the training data includes multiple unimodal data and the multimodal learning model includes a fusion network and a task network;

[0009] Obtain the byte sequence corresponding to each single-modal data in the training data, input each byte sequence into the multimodal learning model, and determine the fusion embedding vector corresponding to all the byte sequences through the fusion network;

[0010] The processing result corresponding to the fusion embedding vector is determined by the task network. Based on the processing result, the network parameters of the fusion network and the task network are adjusted. The process is then repeated to input each byte sequence into the multimodal learning model until the preset iteration termination condition is met, thus obtaining the trained multimodal learning model.

[0011] Optionally, the step of determining the fusion embedding vector corresponding to all the byte sequences through the fusion network includes:

[0012] An embedding vector is generated for each byte sequence through a fusion network, and each embedding vector is fused to obtain a fused embedding vector.

[0013] Optionally, the step of fusing each of the embedding vectors to obtain a fused embedding vector includes:

[0014] Determine the attention weight corresponding to each type of single-modal data;

[0015] The fused embedding vector is obtained by weighting and summing all the embedding vectors based on the attention weights corresponding to each type of single-modal data.

[0016] Optionally, the fusion network includes a linear layer, a computation layer, a normalization layer, and a fusion layer. The step of determining the attention weight corresponding to each type of single-modal data, and performing a weighted summation of all the embedding vectors based on the attention weight corresponding to each type of single-modal data to obtain the fused embedding vector includes:

[0017] The linear layer determines the query vector and key vector corresponding to each embedding vector.

[0018] The computational layer calculates the attention score for each type of single-modal data based on the query vector and key vector corresponding to each embedding vector.

[0019] The attention scores are normalized by the normalization layer to obtain the attention weights corresponding to each type of single-modality data.

[0020] The fusion layer performs a weighted summation of all the embedding vectors based on the attention weights corresponding to each single-modal data to obtain the fused embedding vector.

[0021] The step of calculating the attention score for each type of single-modal data based on the query vector and key vector corresponding to each embedding vector through the computation layer includes:

[0022] For each type of single-modal data, the embedding vector is calculated by the computation layer, and the inner product between the query vector and the key vector corresponding to the embedding vector is used as the attention score for the single-modal data.

[0023] Optionally, after obtaining the trained multimodal learning model, the method further includes:

[0024] If a new modality data training request is received, new training data is obtained based on the new modality data training request, wherein the new training data includes at least new modality data;

[0025] Obtain the byte sequence corresponding to the newly added training data, and input the byte sequence into the multimodal learning model to perform incremental training on the multimodal learning model using the byte sequence corresponding to the newly added single training data.

[0026] Furthermore, to achieve the above objectives, this application also provides a multimodal joint learning device, the multimodal joint learning device comprising:

[0027] An acquisition module is used to acquire training data and a multimodal learning model to be trained, wherein the training data includes multiple unimodal data and the multimodal learning model includes a fusion network and a task network;

[0028] The input module is used to obtain the byte sequence corresponding to each single-modal data in the training data, input each byte sequence into the multimodal learning model, and determine the fusion embedding vector corresponding to all the byte sequences through the fusion network;

[0029] The training module is used to determine the processing result corresponding to the fusion embedding vector through the task network, adjust the network parameters of the fusion network and the task network based on the processing result, and return to execute the step of inputting each byte sequence into the multimodal learning model until the preset iteration termination condition is met, so as to obtain the trained multimodal learning model.

[0030] In addition, to achieve the above objectives, this application also provides a multimodal joint learning device, including: a memory, a processor, and a multimodal joint learning program stored in the memory and executable on the processor, wherein the multimodal joint learning program, when executed by the processor, implements the steps of the multimodal joint learning method as described above.

[0031] In addition, to achieve the above objectives, this application also provides a readable storage medium, which includes a computer-readable storage medium storing a multimodal joint learning program. When the multimodal joint learning program is executed by a processor, it implements the steps of the multimodal joint learning method as described above.

[0032] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a multimodal joint learning program. When the multimodal joint learning program is executed by a processor, it implements the steps of the multimodal joint learning method as described above.

[0033] This application acquires training data and a multimodal learning model to be trained. The training data includes multiple unimodal data, and the multimodal learning model includes a fusion network and a task network. The application acquires byte sequences corresponding to each type of unimodal data in the training data, inputs each byte sequence into the multimodal learning model, determines fusion embedding vectors corresponding to all byte sequences through the fusion network, determines the processing result corresponding to the fusion embedding vectors through the task network, adjusts the network parameters of the fusion network and the task network based on the processing result, and returns to execute the step of inputting each byte sequence into the multimodal learning model until a preset iteration termination condition is met, resulting in a trained multimodal learning model. Therefore, compared with the joint learning method of pre-establishing a vocabulary, querying the vocabulary to convert data of different modalities into tokens, and then using these tokens to train a deep learning model, the embodiments of this application directly obtain the byte sequence of each single modality data, input the byte sequence into the multimodal learning model, and use the byte sequence to train the multimodal learning model to achieve joint learning of the multimodal learning model. There is no need for the complex data preprocessing operation of converting data into tokens, thus eliminating the need to build a vocabulary that occupies storage resources and query the vocabulary to allocate tokens that occupies computing resources. The multimodal learning model is directly trained based on the byte sequence to achieve joint learning of the multimodal learning model, thereby reducing the computer system resource consumption required to achieve multimodal joint learning. Attached Figure Description

[0034] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings.

[0035] Figure 1 This is a schematic diagram of the terminal / system structure of the hardware operating environment involved in the embodiments of this application;

[0036] Figure 2 This is a flowchart illustrating the first embodiment of the multimodal joint learning method of this application;

[0037] Figure 3 This is a schematic diagram of the training process architecture for the multimodal joint learning method in this application;

[0038] Figure 4 This is a schematic diagram illustrating the fusion of embedding vectors in the multimodal joint learning method of this application;

[0039] Figure 5 This is a schematic diagram of the byte prediction network structure of the multimodal learning model in the multimodal joint learning method of this application;

[0040] Figure 6 This is a schematic diagram of the device structure of the multimodal joint learning device of this application. Detailed Implementation

[0041] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0042] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a multimodal joint learning device in the hardware operating environment involved in the embodiments of this application.

[0043] like Figure 1 As shown, the multimodal joint learning device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage system independent of the aforementioned processor 1001.

[0044] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the multimodal joint learning device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0045] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and a multimodal joint learning program.

[0046] exist Figure 1 In the multimodal joint learning device shown, the network interface 1004 is mainly used for data communication with other devices; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and memory 1005 in the multimodal joint learning device of this application can be set in the multimodal joint learning device, and the multimodal joint learning device calls the multimodal joint learning program stored in the memory 1005 through the processor 1001 and executes the multimodal joint learning method provided in the embodiments of this application.

[0047] Existing multimodal joint learning methods mainly focus on converting data from different modalities into tokens, and then using these tokens to train deep learning models. These methods typically require complex preprocessing steps, including vocabulary building and token allocation.

[0048] Multimodal joint learning strategies primarily focus on a fixed number of modalities. When new modalities need to be introduced, extensive modifications and fine-tuning of the model are usually required. This approach lacks flexibility and may encounter difficulties in practical applications, especially in scenarios where the number of modalities frequently changes.

[0049] Furthermore, most existing multimodal joint learning methods primarily focus on learning a single task. Although some methods attempt to perform multi-task learning, they often fail to balance the weights among the tasks well, resulting in poor learning performance on some tasks.

[0050] In summary, existing multimodal joint learning methods suffer from at least the following problems in handling dynamic modalities, tokenless fusion, and multi-task learning:

[0051] 1. Reliance on Token Representation: Traditional multimodal joint learning methods rely on tokenized data representation, which requires complex preprocessing, such as vocabulary building and token allocation, thus increasing the complexity of data preparation.

[0052] 2. Limitations of fixed modalities: Most multimodal joint learning methods can only handle a predefined fixed number of modalities. When a new modality is introduced, these methods usually require significant modifications and retraining, lacking flexibility.

[0053] 3. Imbalance in multi-task learning: Multimodal joint learning methods cannot effectively balance the weights among various tasks, which may lead to performance degradation in some tasks.

[0054] Based on the above issues, please refer to Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the multimodal joint learning method of this application. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0055] In this embodiment, the multimodal joint learning method of this application is applied to the aforementioned terminal device. It should be understood that, based on different design needs of practical applications, the multimodal joint learning method of this application can, of course, be applied to other terminal devices in different feasible implementations.

[0056] In this embodiment, as Figure 2 As shown, the multimodal joint learning method includes the following steps:

[0057] Step S10: Obtain training data and a multimodal learning model to be trained, wherein the training data includes multiple single-modal data and the multimodal learning model includes a fusion network and a task network;

[0058] The multimodal learning model to be trained is used to indicate the model to be subjected to multimodal joint learning, which may be a deep learning model. The training data is used to pre-train the multimodal learning model so that the multimodal learning model can complete the multimodal joint learning process.

[0059] The training data includes, but is not limited to, various single-modal data such as text data, image data, audio data, and video data, and may also include data of other modalities, such as sensor data. This embodiment does not impose specific limitations on this.

[0060] Step S20: Obtain the byte sequence corresponding to each single-modal data in the training data, input each byte sequence into the multimodal learning model, and determine the fusion embedding vector corresponding to all the byte sequences through the fusion network;

[0061] Understandably, for data of any modality, computers store it as a sequence of binary bytes. Based on this, the binary byte sequence of each type of single-modality data can be obtained.

[0062] Furthermore, considering that different encoding methods may lead to encoding conflicts, such as using different binary byte sequences to encode the same character, or using the same binary byte sequence to encode different characters, the unimodal data can be converted to a preset encoding format (e.g., text data to UTF-8 encoding, image data to JPEG encoding, audio data to WAV encoding, and video data to H.264 encoding) before obtaining the byte sequence corresponding to the converted unimodal data. This unifies the byte sequence format and avoids encoding conflicts. The preset encoding format can be different for each type of unimodal data; users can pre-set the encoding format for each type of unimodal data and convert the unimodal data based on the preset encoding format.

[0063] In one possible implementation, for text data (whether Chinese, English, etc.), it can be uniformly converted into a UTF-8 encoded byte sequence using the Unicode character set. For image data (JPEG or other image formats), its binary content can be read directly. For audio data (WAV or other audio formats), the processing method is similar to that for images; the binary content of the audio file can be read directly. For video data (video modality), especially complex encoding formats such as H.264, its binary content can also be read directly. The resulting byte sequence will contain all the information of the video, including audio and metadata. For sensor data (digital sensor data), the directly read data is usually time-series data, which can be floating-point numbers, integers, etc.; for analog sensor data, it can first be converted into a digital signal by a digital-to-analog converter, and then read in the same way as digital sensor data. For infrared data: Infrared data is usually represented as an image or heat map, showing the infrared radiation intensity or temperature of different areas. This data can usually be saved as an image format. For infrared data that is already in an image format, such as PNG or JPEG, the binary content of the infrared image can be read directly. If the infrared data is the raw intensity or temperature reading, it can be converted into an image first (for example, using color coding to represent different temperature levels), and then saved as an image format before reading the binary content of the image.

[0064] For example, suppose the training data includes text data: "The cat is on the mat," and image data: such as... Figure 3 The image and audio data of the cat on the mat are shown below. Figure 3For the audio shown, the text data is converted into a byte sequence encoded in UTF-8. Specifically, the Unicode code point of the Chinese character "猫" is U+732B, which is encoded in UTF-8 using three bytes according to the three-byte format of UTF-8 as follows: 11100111 10110010 10101011. The hexadecimal representations of these three bytes are E7, B2, and AB respectively. Therefore, the byte representation of the Chinese character "猫" under UTF-8 encoding is E7 B2 AB (the actually read byte sequence is 11100111 10110010 10101011); similarly, the byte representation of the Chinese character "在" under UTF-8 encoding is E5 B7 A8; "垫": E5 9E AB; "子": E5 AD 90. Image data: converted into a byte sequence in the JPEG (or other preset encoding format) file format. Audio: converted into a byte sequence in WAV (or other preset encoding format).

[0065] After obtaining the byte sequence of each type of unimodal data, the byte sequence of each type of unimodal data is input into the multimodal learning model, and the fused embedding vector is determined through the fusion network of the multimodal learning model. Specifically, the fusion network may convert the byte sequence of each type of unimodal data into an embedding vector, and fuse the embedding vectors corresponding to each type of unimodal data to obtain the fused embedding vector.

[0066] In step S30, the processing result corresponding to the fused embedding vector is determined through the task network, the network parameters of the fusion network and the task network are adjusted based on the processing result, and the step of inputting each said byte sequence into the multimodal learning model is returned to be executed until a preset iteration termination condition is satisfied, so as to obtain a trained multimodal learning model.

[0067] The network parameters are used to indicate all parameters included in the fusion network and the task network, such as weights, biases, etc. As one of the implementation manners, the parameter gradient of each network parameter may be calculated, and the network parameters may be updated based on the gradient descent method.

[0068] The fused embedding vector is processed by the task network to obtain a corresponding processing result. The task network may specifically include one or more neural network layers. It can be understood that the processing manner of the task network for the fused embedding vector is related to the training task. Exemplarily, referring to Figure 3 as shown, when the training task includes text sentiment classification, the task network may output a sentiment label as the processing result through a fully connected layer and a Softmax function; when the training task includes image classification, the task network may output the target existence probability through a fully connected layer and a Sigmoid function( Figure 3The probability of a cat's presence shown in the figure is used as the processing result. When the training task includes audio regression, the network can output the volume level as the processing result through a fully connected layer.

[0069] The training task is a pre-set task, which can be multi-task (with multiple processing tasks, intuitively represented by multiple outputs) or single-task, so that the trained multimodal learning model can perform multi-task processing. For example, refer to... Figure 3 As shown, the training tasks are multi-task, specifically including text classification (a fully connected layer and a Softmax function outputting sentiment labels), image classification (a fully connected layer and a sigmoid function outputting the probability of a cat's presence), and audio regression (a fully connected layer outputting volume levels). The model training process can be as follows: Figure 3 As shown, each modal data in the training data is converted into a corresponding byte sequence, such as text data into text bytes, image data into image bytes, and audio data into audio bytes. These byte sequences are then input into the model, where a fusion network performs tokenless multimodal fusion of the byte sequences to obtain a fused embedding vector. This fusion network includes... Figure 3 The diagram illustrates the byte-wide and local transforms and the modality-weighted average transform. The modality-weighted average transform outputs a fused embedding vector, which is then input into the task network. The task network outputs the processing result, thereby training the model based on the input byte sequence. Specifically, the byte-wide and local transforms are used for byte prediction, and the modality-weighted average transform is used to calculate the fused embedding vector.

[0070] Furthermore, this training task can be further divided into unimodal tasks and cross-modal tasks. Cross-modal tasks can be used to indicate that there are multiple unimodal data in the input and output (such as input text data and output image data; input text data and audio data, output audio data, etc.), while unimodal tasks are used to indicate that there is only one type of unimodal data in the input and output (such as input text data and output text data; input image data and output image data). For example, refer to... Figure 4As shown, assuming the data is divided into three modalities: text (such as text data), vision (such as infrared data, image data, video data, etc.), and speech (such as audio data), then the training tasks corresponding to different combinations of these three modalities include machine translation, text-to-image generation, speech synthesis, visual description, image search, image-to-speech generation, speech recognition, speech-to-image generation, and multilingual translation. By training a multimodal learning model with different modalities of data, efficient collaboration of text, vision, speech, and other multimodal data can be achieved.

[0071] Furthermore, before training the multimodal learning model based on the fused embedding vector, the multimodal learning model can also be trained to predict byte sequences based on the byte sequences of each single-modal data. That is, training the byte sequence prediction function of the multimodal learning model enables it to predict the next byte sequence based on the current byte sequence. For example, refer to... Figure 5 As shown, the multimodal learning model includes a byte sequence prediction network, which comprises a global network layer ( Figure 5 The Global Model and Local Network Layers shown in the figure Figure 5 The LocalModel shown in the image uses its own sequence encoding ( Figure 5 The Patch Embedded data, as shown, is converted into a byte sequence and used as input to the global network layer. The output of the global network layer serves as input to the local network layer, which then outputs the final prediction result. The global network layer predicts the byte sequence, while the local network layer predicts individual bytes. For example, refer to... Figure 5 As shown, assuming the initial byte sequence predicted by the global network layer is E7B2AB, the local network layer predicts the actual meaning of E7B2AB, such as "cat". Similarly, if the global network layer predicts E5B7A8, the local network layer predicts the actual meaning of E5B7A8, such as "at". If the global network layer predicts E59EAB, the local network layer predicts the actual meaning of E59EAB, such as "pad". If the global network layer predicts E5AD90, the local network layer predicts the actual meaning of E5AD90, such as "child". Therefore, based on the actual input byte sequence and its corresponding data representation, the network parameters of the global and local network layers are continuously adjusted, enabling the multimodal learning model to complete the learning and training of byte sequence prediction.

[0072] In this embodiment, training data and a multimodal learning model to be trained are obtained. The training data includes multiple single-modal data, and the multimodal learning model includes a fusion network and a task network. The byte sequence corresponding to each single-modal data in the training data is obtained, and each byte sequence is input into the multimodal learning model. The fusion network determines the fusion embedding vector corresponding to all the byte sequences. The task network determines the processing result corresponding to the fusion embedding vector, and the network parameters of the fusion network and the task network are adjusted based on the processing result. The process of inputting each byte sequence into the multimodal learning model is repeated until a preset iteration termination condition is met, resulting in a trained multimodal learning model. Thus, compared to the joint learning approach that involves pre-establishing a vocabulary, querying the vocabulary to convert data from different modalities into tokens, and then using these tokens to train a deep learning model, this embodiment directly obtains the byte sequence of each single-modal data, inputs the byte sequence into the multimodal learning model, and uses this byte sequence to train the multimodal learning model, thereby achieving joint learning of the multimodal learning model. This eliminates the need for the complex data preprocessing operation of converting data into tokens, thus avoiding the need to build a vocabulary that occupies storage resources and query the vocabulary to allocate tokens that consume computing resources. By directly training the multimodal learning model based on the byte sequence, joint learning of the multimodal learning model is achieved, thereby reducing the computer system resource consumption required to implement multimodal joint learning.

[0073] Furthermore, based on the first embodiment of this application described above, a second embodiment of the multimodal joint learning method of this application is proposed. Contents identical or similar to those in the first embodiment can be referred to the above description and will not be repeated hereafter. In this embodiment, the step of determining the fused embedding vector corresponding to all the byte sequences through the fusion network includes:

[0074] Step A10: Generate an embedding vector corresponding to each byte sequence through a fusion network, and fuse each embedding vector to obtain a fused embedding vector.

[0075] Specifically, the fusion network may include a vector extraction subnetwork and a Transformer subnetwork. The vector extraction subnetwork generates an embedding vector corresponding to each byte sequence, and the Transformer subnetwork fuses each embedding vector to obtain a fused embedding vector.

[0076] As one implementation method, the vector extraction sub-network can generate the embedding vector corresponding to the byte sequence by word embedding. As another implementation method, the vector extraction sub-network can be any pre-set model network, such as the CNCLIP model network (Chinese Contrastive Language-Image Pre-Training), the Transformer model network, etc.

[0077] It should be noted that each type of unimodal data is converted into an embedding vector of the same dimension to unify the vector dimension of the embedding vectors corresponding to all unimodal data, which facilitates subsequent data processing. For example, if the preset is to convert each type of modal data into a 512-dimensional embedding vector, then for text data, image data, and audio data, their embedding vectors can be represented as follows: text embedding vector: [e1, e2...e512], image embedding vector: [i1, i2...i512], and audio embedding vector: [a1, a2...a512].

[0078] In the Transformer subnetwork, a fusion rule for the embedding vectors can be preset, and the embedding vectors can be fused based on the fusion rule. For example, the fused embedding vector can be obtained by aggregating the embedding vectors, or by weighted summation of the embedding vectors.

[0079] In this embodiment, an embedding vector corresponding to each byte sequence is generated by a fusion network, and each embedding vector is fused to obtain a fused embedding vector. All byte sequences are fused into a vector, which facilitates subsequent data processing.

[0080] In one possible implementation, the step of fusing each of the embedding vectors to obtain a fused embedding vector includes:

[0081] Step B10: Determine the attention weights corresponding to each type of single-modal data;

[0082] Step B20: Based on the attention weights corresponding to each type of single-modal data, all the embedding vectors are weighted and summed to obtain the fused embedding vector.

[0083] In one implementation, the attention weights can be used as network parameters of a multimodal learning model and iteratively updated during training. For example, the correspondence between different unimodal data and attention weights can be seen in Table 1. Based on the correspondence shown in Table 1, the attention weights for each type of unimodal data in the training data can be obtained. As the attention weights are iteratively updated as network parameters during training, Table 1 is also iteratively updated during model training.

[0084] Modal Attention weight Text data We Image data Wi Audio data Wa Video data Wh Sensor data Ws Infrared data Wf

[0085] Table 1

[0086] For example, assuming the training data includes text data, image data, and audio data, with corresponding embedding vectors of [e1, e2...e512], [i1, i2...i512], and [a1, a2...a512], and corresponding attention weights of We, Wi, and Wa, then the fused embedding vector is [We*e1+Wi*i1+Wa*a1, We*e2+Wi*i2+Wa*a2...We*e512+Wi*i512+Wa*a512]. The coordinate representation of the fused embedding vector is shown below. Figure 4 As shown, Figure 4 The image shows the fused embedding vectors for the text, image, and audio of "a puppy," and the fused embedding vectors for the text, image, and audio of "a river."

[0087] In this embodiment, all embedding vectors are weighted and summed based on attention weights to obtain a fused embedding vector. This allows for the training of appropriate attention weights for each type of single-modal data during model training, enabling adaptive adjustment of attention weights and facilitating the fusion of feature vectors from cross-modal data in multimodal learning.

[0088] Furthermore, based on the first and second embodiments of this application described above, a third embodiment of the multimodal joint learning method of this application is proposed. Contents identical or similar to the first or second embodiments described above can be referred to the above description and will not be repeated hereafter. In this embodiment, the fusion network includes a linear layer, a computational layer, a normalization layer, and a fusion layer. The step of determining the attention weights corresponding to each type of single-modal data, and performing a weighted summation of all the embedding vectors based on the attention weights corresponding to each type of single-modal data to obtain the fused embedding vector includes:

[0089] Step C10: Determine the query vector and key vector corresponding to each embedding vector through the linear layer;

[0090] The fusion network may specifically include a Transformer subnetwork, which includes a linear layer, a computation layer, a normalization layer, and an output layer. The linear layer of the weight computation unit converts the embedding vector into a query vector Q, a key vector K, and a value vector V.

[0091] Step C20: The computational layer calculates the attention score for each type of single-modal data based on the query vector and key vector corresponding to each embedding vector.

[0092] The step of calculating the attention score for each type of single-modal data based on the query vector and key vector corresponding to each embedding vector through the computation layer includes:

[0093] Step D10: For each type of single-modal data corresponding to the embedding vector, the computation layer calculates the inner vector product between the query vector and the key vector corresponding to the embedding vector, and uses the inner vector product as the attention score corresponding to the single-modal data.

[0094] In the computational layer, for each query vector Q of a single modality, the dot product of Q with the key vectors K of all modalities is calculated to obtain the attention score. Specifically, the dot product between the query vector Q of each single modality and its own key vector K can be used as the attention score for that single modality.

[0095] Step C30: Normalize all the attention scores through the normalization layer to obtain the attention weights corresponding to each type of single-modality data;

[0096] Specifically, the Softmax function can be applied to the normalization layer for normalization. That is, the attention scores obtained by the normalization layer are normalized by the Softmax function, so that a probability distribution can be obtained, which represents the importance of each key vector K relative to the query vector Q.

[0097] Step C40: The fusion layer performs a weighted summation of all the embedding vectors based on the attention weights corresponding to each single-modal data to obtain the fused embedding vector.

[0098] In this embodiment, the importance of each modality in the fusion is dynamically calculated, and the embeddings of each modality are weighted using normalized attention weights. This mechanism allows the model to dynamically adjust its attention to different modalities based on the current data content and context. It is particularly suitable for newly introduced modalities without requiring major changes to the model structure; only the corresponding attention calculation module needs to be added. This achieves adaptive adjustment and calculation of attention weights, improving the convenience of adding new modal data. Furthermore, the adaptive adjustment and calculation of attention weights can be achieved through the Transformer network. Using a multi-scale Transformer network structure, the model can better capture the multi-scale features of each modality's data, improving the model's accuracy and robustness, thereby ensuring the accuracy of the attention weights in assessing the importance of each modality's data.

[0099] Furthermore, to aid in understanding the technical concept or working principle of attention weight calculation in this embodiment, a specific embodiment is provided. In this specific embodiment, the calculation process of attention weight is as follows:

[0100] 1. Generate Q, K, V vectors: For each modality (text, image, audio, etc.), the embedding vector of the modality is transformed through a linear layer to obtain Q, K, V.

[0101] 2. Calculate the attention score: For a query Q of a specific modality, calculate its dot product with all modality keys K to obtain the attention score.

[0102] 3. Apply the Softmax function: Normalize the obtained attention scores using the Softmax function to obtain a probability distribution representing the importance of each key K relative to the query Q.

[0103] 4. Obtaining Attention Weights: For a specific modality, this probability distribution (the output of Softmax) can be viewed as the weights of interactions between modalities. For example, if we calculate the attention score of a text modality relative to itself, the output of Softmax can be used as the attention weights We for the text data.

[0104] 5. Integrate weights from different modalities:

[0105] - Apply the obtained attention weights We, Wi, Wa to the embedding value V of the corresponding modality.

[0106] - Weighted summation: We multiplied by the text's V, Wi multiplied by the image's V, Wa multiplied by the audio's V.

[0107] Then add them together to get the final fused embedding vector.

[0108] The above steps apply to any number of modalities, including newly added modalities. Simply generate new Q, K, and V values ​​for each new modality and calculate the attention score and weights using the same process.

[0109] For example, if a new video modality V is added, Qv, Kv, and Vv will be generated for it, and Wv will be calculated using dot product and Softmax. Then, all weights are normalized using Softmax (We, Wi, Wa, Wv) to ensure that their sum is 1.

[0110] It should be noted that the above specific embodiments are only used to understand this application and do not constitute a limitation on the attention weight calculation process of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0111] Furthermore, based on the first, second, and third embodiments of this application described above, a fourth embodiment of the multimodal joint learning method of this application is proposed. Contents identical or similar to the first, second, or third embodiments described above can be referred to the above description and will not be repeated hereafter. In this embodiment, after the step of obtaining the trained multimodal learning model, the method further includes:

[0112] Step E10: If a new modality data training request is received, new training data is obtained based on the new modality data training request, wherein the new training data includes at least new modality data;

[0113] The newly added modal data can be either single-modal or multi-modal data. For example, if the previous training of the multi-modal learning model used text data, image data, and audio data, and now video data is added, then the video modal data is the newly added modal data. The newly added training data must include at least the newly added modal data. For example, the newly added modal data and the training data from the previous training of the video multi-modal learning model can be used as the newly added training data.

[0114] Step E20: Obtain the byte sequence corresponding to the newly added training data, and input the byte sequence into the multimodal learning model to perform incremental training on the multimodal learning model using the byte sequence corresponding to the newly added single training data.

[0115] Understandably, during incremental training, the attention score of the newly added modality data can be calculated based on the computational layer of the fusion network. Specifically, the byte sequence corresponding to the newly added modality data can be input into the fusion network, and the new attention score of the newly added modality data can be calculated through the computational layer of the fusion network. Attention scores of other existing modality data before the newly added modality data are obtained and used as historical attention scores. Thus, only the attention score of the newly added modality data needs to be calculated, and there is no need to repeatedly calculate the attention scores of existing modality data, thereby avoiding the duplication of attention score calculation.

[0116] The newly added attention scores and historical attention scores are normalized to obtain the latest attention weights for each modality.

[0117] The incremental training can specifically include byte sequence prediction training and training task training. Byte sequence prediction training can be based on the byte sequences corresponding to the newly added modality data. The specific training method is similar to that of historical modality data byte sequence prediction training, and will not be repeated here. For example, a pre-training objective function Jpretrain can be defined, and the multimodal learning model can be trained based on minimizing the objective of this objective function. For example, setting... Where K is the number of training data points, and L(k) is the loss at each data point, such as the deviation between the actual byte sequence and the predicted byte sequence. For training tasks, training tasks can be set in advance, and the fusion embedding vectors corresponding to the training data in the training task can be determined based on the updated attention weights. Incremental training of the multimodal learning model can then be completed based on the transformed fusion embedding vectors.

[0118] This embodiment proposes an incremental introduction and training strategy for new modalities. When introducing a new modality, no major changes to the overall model structure are required. By combining pre-training of the new modality with fine-tuning of the overall model, the model can effectively integrate knowledge from the new modality while maintaining the performance of existing modalities.

[0119] Furthermore, the training steps for a multimodal learning model may include:

[0120] Step F10: Obtain the training task and the loss function corresponding to each training task;

[0121] As one implementation method, a loss function corresponding to each type of training task can be pre-set, determining the corresponding loss function based on the task category to which each training task belongs. For example, for classification tasks, cross-entropy loss can be set as the loss function, specifically... Where y is the real label. Here, N represents the predicted label of the model, N is the total number of training data points included in the i-th training task, task represents the i-th training task, and j represents the j-th training data point. For regression tasks, the mean squared error can be set as the loss function, specifically as follows: Where M is the total number of training data included in the i-th type of training task.

[0122] Among them, training tasks with the same training objective are classified as training tasks, such as generating an audio clip based on input text and an image, generating an image based on input text, translating input text, generating an audio clip based on input text, and continuing text based on input text, etc.

[0123] Step F20: Determine the weighted sum loss function of all the loss functions, and use the weighted sum loss function as the objective function of the multimodal learning model;

[0124] Let the task weight of the i-th training task be α. i Then the objective function can be: Where T is the total number of tasks, the objective function of the multimodal learning model is minimized by weighted sum loss function, thereby ensuring that all training tasks receive balanced attention.

[0125] Step F30: The multimodal learning model is trained based on the objective function and the byte sequence input into the multimodal learning model.

[0126] During training, an objective function is defined so that the multimodal learning model iteratively updates in the direction of minimizing the objective function value, ensuring that the multimodal learning model after incremental training can take into account the processing capabilities of each task and better balance the needs of multiple tasks.

[0127] In this embodiment, by employing a dynamic combination of loss functions, the model is allowed to automatically adjust the weights of the loss function for each task based on the relevance of each task and its current performance. This means that the model no longer simply uses the same loss function weights for all tasks, but can adaptively adjust them according to the learning progress and difficulty of each task. Furthermore, through this dynamic adjustment mechanism, the model can reduce interference with other tasks while performing one task. For example, if a task performs poorly at a certain stage, the model will automatically increase the weight of its loss function, thereby increasing its focus on that task. This strategy ensures that the model achieves better balance and generalization ability in multi-task learning. Moreover, this strategy of combining and adjusting loss functions not only improves the learning efficiency of each task but also optimizes the overall performance of the entire multi-task model. This is because the model can more intelligently allocate its learning resources, ensuring that all tasks receive appropriate training and optimization.

[0128] Furthermore, for each specific training task (e.g., text classification, image classification, etc.), a task-specific network head can be pre-designed based on the fused embedding vectors. For each specific task, a task-specific function ftaski can be pre-defined to extract information from the fused embeddings and generate task-specific outputs, thus achieving balanced multi-task learning.

[0129] In addition, this application also provides a multimodal joint learning device, referring to Figure 6 The multimodal joint learning device includes:

[0130] The acquisition module 10 is used to acquire training data and a multimodal learning model to be trained, wherein the training data includes multiple single-modal data and the multimodal learning model includes a fusion network and a task network;

[0131] Input module 20 is used to obtain the byte sequence corresponding to each single-modality data in the training data, input each byte sequence into the multimodal learning model, and determine the fusion embedding vector corresponding to all the byte sequences through the fusion network;

[0132] The training module 30 is used to determine the processing result corresponding to the fusion embedding vector through the task network, adjust the network parameters of the fusion network and the task network based on the processing result, and return to execute the step of inputting each byte sequence into the multimodal learning model until the preset iteration termination condition is met, so as to obtain the trained multimodal learning model.

[0133] The training module 30 is also used for:

[0134] An embedding vector is generated for each byte sequence through a fusion network, and each embedding vector is fused to obtain a fused embedding vector.

[0135] The training module 30 is also used for:

[0136] Determine the attention weight corresponding to each type of single-modal data;

[0137] The fused embedding vector is obtained by weighting and summing all the embedding vectors based on the attention weights corresponding to each type of single-modal data.

[0138] The fusion network includes a linear layer, a computational layer, a normalization layer, and a fusion layer. The training module 30 is further used for:

[0139] The linear layer determines the query vector and key vector corresponding to each embedding vector.

[0140] The computational layer calculates the attention score for each type of single-modal data based on the query vector and key vector corresponding to each embedding vector.

[0141] The attention scores are normalized through the normalization layer to obtain the attention weights corresponding to each type of single-modality data.

[0142] The fusion layer performs a weighted summation of all the embedding vectors based on the attention weights corresponding to each single-modal data to obtain the fused embedding vector.

[0143] The training module 30 is also used for:

[0144] For each type of single-modal data, the embedding vector is calculated by the computation layer, and the inner product between the query vector and the key vector corresponding to the embedding vector is used as the attention score for the single-modal data.

[0145] The multimodal joint learning device further includes an incremental training module, which is used for:

[0146] If a new modality data training request is received, new training data is obtained based on the new modality data training request, wherein the new training data includes at least new modality data;

[0147] Obtain the byte sequence corresponding to the newly added training data, and input the byte sequence into the multimodal learning model to perform incremental training on the multimodal learning model using the byte sequence corresponding to the newly added single training data.

[0148] Furthermore, this application also proposes a multimodal joint learning device, which includes a memory, a processor, and a multimodal joint learning program stored in the memory and executable on the processor. When the multimodal joint learning program is executed by the processor, it implements the steps of the multimodal joint learning method as described above.

[0149] The specific implementation of the multimodal joint learning device in this application is basically the same as the embodiments of the multimodal joint learning method described above, and will not be repeated here.

[0150] In addition, to achieve the above objectives, this application also provides a readable storage medium storing a multimodal joint learning program, which, when executed by a processor, implements the steps of the multimodal joint learning method as described above.

[0151] The specific implementation of the readable storage medium in this application is basically the same as the embodiments of the multimodal joint learning method described above, and will not be repeated here.

[0152] Furthermore, embodiments of this application also propose a computer program product, including a multimodal joint learning program, which, when executed by a processor, implements the steps of the multimodal joint learning method as described above.

[0153] The specific implementation of the computer program product in this application is basically the same as the embodiments of the multimodal joint learning method described above, and will not be repeated here.

[0154] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0155] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0156] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0157] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A multimodal joint learning method, characterized in that, The multimodal joint learning method includes the following steps: Acquire training data and a multimodal learning model to be trained, wherein the training data includes multiple single-modal data, and the multimodal learning model includes a fusion network and a task network, wherein the fusion network includes a linear layer, a computational layer, a normalization layer and a fusion layer; Obtain the byte sequence corresponding to each single-modal data in the training data, input each byte sequence into the multimodal learning model, generate the embedding vector corresponding to each byte sequence through the fusion network, and determine the query vector and key vector corresponding to each embedding vector through the linear layer; The computational layer calculates the attention score for each type of single-modal data based on the query vector and key vector corresponding to each embedding vector. The attention scores are normalized through the normalization layer to obtain the attention weights corresponding to each type of single-modality data. The fusion layer performs a weighted summation of all the embedding vectors based on the attention weights corresponding to each type of single-modal data to obtain a fused embedding vector; The processing result corresponding to the fusion embedding vector is determined by the task network. Based on the processing result, the network parameters of the fusion network and the task network are adjusted. The process is then repeated to input each byte sequence into the multimodal learning model until the preset iteration termination condition is met, thus obtaining the trained multimodal learning model.

2. The multimodal joint learning method as described in claim 1, characterized in that, The step of calculating the attention score for each type of single-modal data based on the query vector and key vector corresponding to each embedding vector through the computation layer includes: For each type of single-modal data, the embedding vector is calculated by the computation layer, and the inner product between the query vector and the key vector corresponding to the embedding vector is used as the attention score for the single-modal data.

3. The multimodal joint learning method as described in any one of claims 1 to 2, characterized in that, After obtaining the trained multimodal learning model, the method further includes: If a new modality data training request is received, new training data is obtained based on the new modality data training request, wherein the new training data includes at least new modality data; Obtain the byte sequence corresponding to the newly added training data, and input the byte sequence into the multimodal learning model to perform incremental training on the multimodal learning model using the byte sequence corresponding to the newly added training data.

4. A multimodal joint learning device, characterized in that, The multimodal joint learning device includes: The acquisition module is used to acquire training data and a multimodal learning model to be trained. The training data includes multiple single-modal data, and the multimodal learning model includes a fusion network and a task network. The fusion network includes a linear layer, a computational layer, a normalization layer, and a fusion layer. The input module is used to obtain the byte sequence corresponding to each single-modal data in the training data, input each byte sequence into the multimodal learning model, generate the embedding vector corresponding to each byte sequence through the fusion network, and determine the query vector and key vector corresponding to each embedding vector through the linear layer; The computational layer calculates the attention score for each type of single-modal data based on the query vector and key vector corresponding to each embedding vector. The attention scores are normalized through the normalization layer to obtain the attention weights corresponding to each type of single-modality data. The fusion layer performs a weighted summation of all the embedding vectors based on the attention weights corresponding to each type of single-modal data to obtain a fused embedding vector; The training module is used to determine the processing result corresponding to the fusion embedding vector through the task network, adjust the network parameters of the fusion network and the task network based on the processing result, and return to execute the step of inputting each byte sequence into the multimodal learning model until the preset iteration termination condition is met, so as to obtain the trained multimodal learning model.

5. A multimodal joint learning device, characterized in that, The multimodal joint learning device includes: a memory, a processor, and a multimodal joint learning program stored in the memory and executable on the processor, wherein the multimodal joint learning program, when executed by the processor, implements the steps of the multimodal joint learning method as described in any one of claims 1 to 3.

6. A readable storage medium, characterized in that, The readable storage medium includes a computer-readable storage medium storing a multimodal joint learning program, which, when executed by a processor, implements the steps of the multimodal joint learning method as described in any one of claims 1 to 3.

7. A computer program product, characterized in that, The computer program product includes a multimodal joint learning program, which, when executed by a processor, implements the steps of the multimodal joint learning method as described in any one of claims 1 to 3.

Citation Information

Patent Citations

  • Learning participation degree identification method, apparatus and device, and readable storage medium

    CN114926716A

  • Smart contract security analysis method and system based on multi-modal technology

    CN116958767A