Multi-modal data learning analysis method and device

Abstract

The invention discloses a multi-modal data learning analysis method and device, and belongs to the technical field of data processing. The method comprises the steps of obtaining a multi-modal data set based on a task demand; counting the occurrence frequency of each mode in the data set in the samples, and determining the quotient of the total number of the samples and the occurrence frequency as a learning speed factor of each mode; calculating a learning capability upper limit and a current learning state of each mode, and determining a difference value between the learning capability upper limit and the current learning state as a learning state factor of each mode; training a multi-modal model by using the plurality of samples, and updating model parameters of the multi-modal model by using a loss function, so as to perform learning analysis on multi-modal data by using the trained multi-modal model; the loss function is constructed by utilizing the learning speed factor and the learning state factor of each mode. According to the method, the learning ability of the multi-modal model for the modals with poor performance can be analyzed and improved, and a more balanced and effective multi-modal learning process is ensured.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multimodal data learning and analysis method and device. Background Art

[0002] Multimodal learning integrates and leverages information from diverse data sources (such as text, images, audio, video, and sensor data) to enable models to more comprehensively understand and process complex tasks. Compared to single-modality learning, multimodal learning leverages the complementarity between different data sources to improve model robustness and performance.

[0003] Currently, the main challenges facing multimodal models are incomplete data modalities and imbalanced data learning. Incomplete data modalities refer to the absence of one or more modalities in a sample due to factors such as sensor failure, high data collection costs, or privacy concerns. Imbalanced data learning refers to the phenomenon in which data from different modalities differ significantly in quantity, quality, or representativeness, resulting in a significant bias in the model's learning of certain modalities. Learning and analyzing using incomplete and imbalanced data typically consumes significant time and resources. Multimodal models are also sensitive to such real-world data, significantly impacting model performance.

[0004] Therefore, there is an urgent need to provide a learning analysis method to ensure that a multimodal model with better performance is learned from incomplete and imbalanced data. Summary of the Invention

[0005] The present invention provides a multimodal data learning and analysis method and device. The technical solution is as follows:

[0006] In one aspect, a multimodal data learning and analysis method is provided, the method comprising:

[0007] Acquire a multimodal dataset based on task requirements; the dataset includes multiple samples, each sample including at least some of all modalities;

[0008] Counting the number of occurrences of each modality in the samples of the data set, and determining the quotient of the total number of samples and the number of occurrences as the learning speed factor of each modality;

[0009] Calculate the learning ability upper limit and current learning state of each modality, and determine the difference between the learning ability upper limit and the current learning state as the learning state factor of each modality;

[0010] The multimodal data of the sample is used as input, and the actual task results of the sample annotation are used as output. The multimodal model is trained, and the model parameters of the multimodal model are updated using the loss function, so that the trained multimodal model can be used to learn and analyze the multimodal data; the loss function is constructed using the learning speed factor and learning state factor of each modality.

[0011] In another aspect, a multimodal data learning and analysis device is provided, the device comprising:

[0012] An acquisition unit, configured to acquire a multimodal dataset based on task requirements; the dataset includes a plurality of samples, each sample including at least some of all modalities;

[0013] a determination unit, configured to count the number of occurrences of each modality in the samples of the data set, and determine the quotient of the total number of samples and the number of occurrences as a learning speed factor for each modality;

[0014] The determining unit is further configured to calculate the learning ability upper limit and the current learning state of each modality, and determine the difference between the learning ability upper limit and the current learning state as the learning state factor of each modality;

[0015] An analysis unit is used to take the multimodal data of the sample as input and the actual task results of the sample annotation as output, train the multimodal model, and update the model parameters of the multimodal model using a loss function, so as to use the trained multimodal model to learn and analyze the multimodal data; the loss function is constructed using the learning speed factor and learning state factor of each modality.

[0016] On the other hand, a computer device is provided, which includes a memory and a processor, wherein the memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory to implement the steps of the multimodal data learning and analysis method described above.

[0017] On the other hand, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of the multimodal data learning and analysis method described above are implemented.

[0018] On the other hand, a computer program product is provided, comprising a computer program, which implements the steps of the multimodal data learning and analysis method described above when executed by a processor.

[0019] The technical solution provided by the present invention can at least bring the following beneficial effects:

[0020] By introducing a learning speed factor, the learning update speed of modalities with a higher presence ratio is reduced, and the learning update speed of modalities with a lower presence ratio is increased, thereby balancing the problem of incomplete sample modalities; by introducing a learning state factor to evaluate the current learning state of each modality, and calculating the gap between the current learning state of each modality and the upper limit of learning ability, the gap can be used to provide different learning incentives to enhance the learning process; it can be seen that this scheme can analyze and improve the learning ability of the multimodal model for modalities with poor performance, ensuring a more balanced and effective multimodal learning process. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 This is a flow chart of a multimodal data learning and analysis method provided by one embodiment of the present invention;

[0023] Figure 2 is a schematic diagram of a multimodal model structure provided by an embodiment of the present invention;

[0024] Figure 3 Schematic diagram of a feature reconstruction network performing a feature completion task according to an embodiment of the present invention;

[0025] Figure 4 is a quantitative analysis result of data set 4 provided by an embodiment of the present invention;

[0026] Figures 5 to 8 These are the qualitative analysis results obtained for four data sets according to an embodiment of the present invention;

[0027] Figure 9 This is a structural diagram of a multimodal data learning and analysis device provided by one embodiment of the present invention;

[0028] Figure 10 This is a hardware architecture diagram of a computer device provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0030] Considering that related technologies for learning and analyzing incomplete and imbalanced data mainly focus on the imbalance and modal convergence speed issues at the dataset level (macro level), relative distance metrics are used to balance the learning and analysis between modalities. However, the contribution differences and learning status of multimodality at the sample level (micro level) are ignored. In real-world scenarios, modal differences at the sample level are common. For example, in "a picture of a bicycle and a text about bicycles" and "a magnified partial picture of a bicycle wheel and a text with bicycle as the subject", the contributions of the image modality and text modality to the target classification task are significantly different. It is very necessary to conduct a more fine-grained analysis of modal differences and formulate targeted optimization strategies.

[0031] The specific implementation of the concept of the present invention is described below.

[0032] Please refer to Figure 1 , an embodiment of the present invention provides a multimodal data learning and analysis method, the method comprising:

[0033] Step 100: Acquire a multimodal dataset based on task requirements; the dataset includes multiple samples, each sample contains at least some of all modalities;

[0034] Step 102: Count the number of occurrences of each modality in the samples of the data set, and determine the quotient of the total number of samples and the number of occurrences as the learning speed factor of each modality;

[0035] Step 104: Calculate the learning capability upper limit and current learning state of each modality, and determine the difference between the learning capability upper limit and the current learning state as the learning state factor of each modality;

[0036] Step 106: The multimodal data of the sample is used as input, and the actual task results of the sample annotation are used as output to train the multimodal model, and the model parameters of the multimodal model are updated using the loss function, so as to use the trained multimodal model to perform learning and analysis on the multimodal data; the loss function is constructed using the learning speed factor and learning state factor of each modality.

[0037] In an embodiment of the present invention, a learning speed factor is introduced to reduce the learning update speed of modalities with a higher existence ratio and to increase the learning update speed of modalities with a lower existence ratio, thereby balancing the problem of incomplete sample modalities; a learning state factor is introduced to evaluate the current learning state of each modality, and the gap between the current learning state of each modality and the upper limit of the learning ability is calculated, so as to utilize the gap for different learning incentives to enhance the learning process; it can be seen that this scheme can analyze and improve the learning ability of the multimodal model for modalities with poor performance, ensuring a more balanced and effective multimodal learning process.

[0038] Described below Figure 1 How to perform the steps shown.

[0039] First, for step 100 , a multimodal dataset is obtained based on task requirements; the dataset includes multiple samples, and each sample contains at least some of all modalities.

[0040] The embodiment of the present invention can be applied to the basic framework of the multimodal model, please refer to Figure 2 , which is a schematic diagram of the basic framework principle of the multimodal model. The multimodal model includes: encoders corresponding to each modality, a feature fusion network connected to each encoder, and a task multimodal decoder connected to the feature fusion network; the multimodal model can use the corresponding encoders to extract features from different modalities, and then collect the features extracted from each modality to perform multimodal feature fusion. Finally, according to the task requirements, the fused multimodal features are analyzed, detected or predicted using the multimodal decoder to obtain the final output result.

[0041] In the embodiments of the present invention, different multimodal datasets are obtained depending on the task requirements. For example, the task requirement is 3D object detection for vehicle-road collaboration, while another example is the task requirement for classifying and detecting animals and plants contained in an image. Multimodal data can include text, images, audio, video, sensor data, etc.

[0042] The dataset in the embodiments of the present invention is an incomplete multimodal dataset. Specifically, the dataset includes multiple samples, each of which contains at least some of the modalities. For example, the dataset contains five modalities: text, image, audio, video, and sensor data. Some samples only include data from four modalities: text, image, audio, and video, while others only include data from three modalities: image, audio, and video.

[0043] Then, step 102 "counting the number of occurrences of each modality in the sample in the data set, and determining the quotient of the total number of samples and the number of occurrences as the learning speed factor of each modality" and step 104 "calculating the learning ability upper limit and current learning state of each modality, and determining the difference between the learning ability upper limit and the current learning state as the learning state factor of each modality" are explained at the same time.

[0044] In this embodiment of the present invention, assuming that the total number of samples in the data set is N, if the number of times mode m appears is n, then the learning speed factor of mode m is: Assuming that the modality m is text, then count whether the text exists in each sample in the statistical data set, and count the number of samples with text as the number of occurrences of the text.

[0045] Since the data set is generally processed in batches for training, after calculating the learning speed factor of each modality, the samples in the data set can also be processed in batches, and the learning state factor can be calculated for each batch of samples.

[0046] In one embodiment of the present invention, the upper limit of the learning ability of each modality is calculated as follows:

[0047] For each modality, the following steps are performed: using a single-modality decoder to perform task-specific decoding on the single modality feature in the sample to predict the output task result, and calculating the accuracy using the predicted output task result and the actual task result annotated by the sample; this accuracy is determined as the upper limit of the learning ability of the modality.

[0048] Assume that the modal characteristic of mode m is h m , the upper limit of learning ability is Add additional single-mode decoder for mode m The unimodal decoder has the same structure as the multimodal decoder in the multimodal model, but different parameters. The unimodal decoder is used to decode the modal features h of the unimodal m. m Perform task-specific decoding to predict the output task result y m , and use the predicted output task result y m and actual task results of sample annotation Calculate the accuracy. The following formula can be used to calculate the accuracy, which is the upper limit of the learning ability of the mode m:

[0049]

[0050] Among them, f acc (·) is the function for calculating accuracy.

[0051] In one embodiment of the present invention, the current learning state of each modality is calculated as follows:

[0052] For each modality, the following steps are performed: freely combining the other modalities in the data set except the modality to obtain multiple first subsets; for each first subset, calculating the contribution value that can be provided to the first subset after adding the modality to the first subset, and determining the sum of the contribution values ​​of the multiple first subsets as the current learning state of the single modality.

[0053] Assume that there are M modalities in the data set. It is a first subset of multiple sets formed by the free combination of all modes except mode m. The feature fusion network in the multimodal model is f fusion (·), the multimodal decoder is f dec (·), then the mode m can be added to the first subset by the following formula The first subset The contribution value provided includes:

[0054]

[0055] in, To add mode m to the first subset The first subset The contribution value provided, The first subset The number of modes in , The first subset The profit function, The first subset The number of modes in b is the total number of samples.

[0056] After calculating the mode m, add the mode m to a first subset The first subset After providing the contribution value, the sum of the contribution values ​​of multiple first subsets can be further calculated using the following formula:

[0057]

[0058] Among them, φ m is the total contribution value, and M1 is the number of the first subset.

[0059] The total contribution value is determined as the current learning state of mode m, and then the learning state factor of mode m is calculated using the following formula

[0060] Finally, for step 106, the multimodal data of the sample is used as input, and the actual task results of the sample annotation are used as output, the multimodal model is trained, and the model parameters of the multimodal model are updated using the loss function, so that the trained multimodal model can be used to learn and analyze the multimodal data.

[0061] In an embodiment of the present invention, in order to ensure that a multimodal model with better performance is learned from incomplete and unbalanced data, a loss function can be constructed using the learning speed factor and learning state factor of each modality, so as to use the loss function to tune the parameters of the multimodal model.

[0062] Specifically, the loss function can be:

[0063]

[0064] It can be seen that the total loss of this loss function is It includes three parts of loss. The following is a comparison of these three parts of loss (the first comparison loss Second contrastive loss And the third contrast loss ) are explained separately.

[0065] First contrastive loss The sample is input into the multimodal model, and the multimodal model predicts the output task result y and the actual task result of the sample annotation Make a loss between them and get the first contrast loss

[0066]

[0067] in, is the contrast loss function;

[0068] Second contrastive loss The unimodal features in the sample are input into their respective unimodal decoders, and the unimodal decoders predict the output task results The actual task results with sample annotation Make a loss between them and get the second contrast loss

[0069]

[0070] in, is the learning speed factor of mode m, is the learning state factor of mode m;

[0071] Third contrast loss After freely combining all the modes in the data set, multiple second subsets are obtained. The modal features in the input are input to the feature fusion network for feature fusion, and the fusion features are input to the multimodal decoder to predict the actual task results of the output task results sample annotations. The loss is made between them, the average of the sum of the losses corresponding to all the second subsets is determined as the loss corresponding to a single sample, and the average of the losses corresponding to all samples is calculated to obtain the third comparison loss

[0072]

[0073] Wherein, M2 is the number of the second subset.

[0074] In this way, the above loss function can be used to train the multimodal model to adjust the network parameters of each network in the modality.

[0075] In an embodiment of the present invention, by introducing a learning speed factor into the loss function, the learning update speed of modalities with a higher presence ratio can be reduced, while the learning update speed of modalities with a lower presence ratio can be increased, thereby balancing the problem of incomplete sample modalities. By introducing a learning state factor into the loss function, it can be used to evaluate the current learning state of each modality and calculate the gap between the current learning state of each modality and the upper limit of learning ability, so as to use this gap to provide different learning incentives to enhance the learning process. In this way, the learning ability of the multimodal model for poorly performing modalities can be analyzed and improved, ensuring a more balanced and effective multimodal learning process.

[0076] It should be noted that the unimodal decoder and the multimodal decoder in the embodiments of the present invention are both task-specific decoders based on task requirements.

[0077] In one embodiment of the present invention, in order to further improve the performance of the multimodal model, the feature fusion network may include: a feature reconstruction network and an information fusion network; then, the multimodal data of the sample is used as input and the actual task results of the sample annotation are used as output to train the multimodal model, which may specifically include: inputting the multimodal data of the sample into the encoder of the corresponding modality to use the encoder to extract features, and inputting the extracted modal features into the feature reconstruction network so that the feature reconstruction network reconstructs and restores the modal features that are not included, and outputting the modal features of all modalities to the information fusion network, so as to use the information fusion network to perform multimodal feature fusion to obtain multimodal fusion features.

[0078] Furthermore, in order to improve the accuracy of the feature reconstruction network in recovering the unincluded modal features, an additional feature completion task can be added to the feature reconstruction network on the basis of the original task. Specifically, the feature reconstruction network reconstructs and recovers the unincluded modal features, including:

[0079] Selecting and discarding some modal features from the modal features included in the sample;

[0080] Reconstructing and restoring modal features not included in the sample and discarded modal features using the modal features included in the sample and the modal features of the remaining modalities in the sample, so as to output modal features of all modalities;

[0081] A fourth contrast loss is constructed using the discarded modal features and the modal features reconstructed and restored for the discarded modality, and the fourth contrast loss is updated to the loss function to update the network parameters of the feature reconstruction network using the updated loss function.

[0082] Please refer to Figure 3 , is a schematic diagram of the feature reconstruction network performing feature completion tasks. The sample contains three modalities: h1, h3, and h4, but does not contain h2. Some modal features are selected and discarded from the modal features contained in the sample. Figure 3 The h3 modal feature is selected and discarded, and the feature reconstruction network uses the modal features included in the sample and the modal features of the remaining modalities in the sample to reconstruct and restore the h2 modal features that are not included and the discarded h3 modal features, so as to output the original h1 modal features, the original h4 modal features, and the reconstructed and restored h2 modal features and h3 modal features. Since the original h3 modal features are known, in order to improve the feature reconstruction and recovery capabilities, the fourth contrast loss is constructed using the original h3 modal features and the reconstructed and restored h3 modal features, and the fourth contrast loss is updated to the loss function, and the network parameters of the feature reconstruction network are updated using the updated loss function, so that the feature reconstruction network learns how to reconstruct and restore the non-included modal features from the included modal features. Among them, the implementation method of updating the fourth contrast loss to the loss function can be that the loss function is the sum of the first contrast loss, the second contrast loss, the third contrast loss, and the fourth contrast loss.

[0083] Furthermore, since the feature completion task also uses incomplete and unbalanced data, a learning rate factor and a learning state factor can be introduced for the feature completion task. That is to say, the fourth contrast loss is calculated as follows: calculate the loss between the discarded modal features in the sample and the modal features reconstructed and restored for the discarded modality, multiply the loss by the learning rate factor and the learning state factor of the discarded modality, and after obtaining the contrast loss of the sample, calculate the sum of the contrast losses of all samples to obtain the fourth contrast loss.

[0084] In an embodiment of the present invention, by adding additional feature completion tasks to the feature reconstruction network, firstly, the feature reconstruction network can be assisted to further reduce information loss, avoid deviations caused by sparsity in the model, and improve the accuracy of downstream tasks; secondly, the feature data of existing modalities can be maximized to reduce data collection and annotation costs; finally, the model's fault tolerance to incomplete modal data can be improved, especially in real-time systems (such as autonomous driving), to ensure input integrity.

[0085] The following experiments illustrate the effects of the multimodal data learning and analysis method according to the embodiment of the present invention.

[0086] Experimental conditions:

[0087] Dataset 1: BraTS2020 dataset, including four modalities: Flair, T1ce, T1, and T2. The task requirement is medical image segmentation. The performance indicator is the Dice score of Whole Tumor (WT), Tumor Core (TC), and Enhancing Tumor (ET). Specifically, it is the average Dice score of 15 combinations of the four modalities: Flair, T1ce, T1, and T2.

[0088] Dataset 2: IEMOCAP dataset, including three modalities: text, audio, and video. The task requirement is sentiment analysis, and the performance indicator is the average accuracy of seven combinations of these three modalities.

[0089] Dataset 3: MSP dataset, including three modalities: text, audio, and video. The task requirement is sentiment analysis, and the performance indicator is the average accuracy of seven combinations of these three modalities.

[0090] Dataset 4: Audiovision-MNIST dataset, including two modalities: image and sound. The task requirement is digit recognition, and the performance indicator is accuracy (Acc).

[0091] Experimental methods:

[0092] Quantitative and qualitative comparative analysis of the embodiments of the present invention and the prior art.

[0093] The results of the quantitative comparative analysis of datasets 1-3 are shown in Tables 1, 2, and 3.

[0094] Table 1: Quantitative comparison results of different modality missing rate settings on the BraTS2020 dataset

[0095]

[0096] Table 2: Comparison of quantitative accuracy results under different missing rate settings on the IEMOCAP dataset. The missing rate settings are: a. (0.2, 0.5, 0.8); b. (0.2, 0.8, 0.5); c. (0.5, 0.2, 0.8); d. (0.5, 0.8, 0.2); e. (0.8, 0.2, 0.5); f. (0.8, 0.5, 0.2). The data in the table are the average accuracy of 7 multimodal combinations.

[0097]

[0098] Table 3: Comparison of the accuracy of different missing rate settings on the MSP dataset. The missing rate settings are: a. (0.2, 0.5, 0.8); b. (0.2, 0.8, 0.5); c. (0.5, 0.2, 0.8); d. (0.5, 0.8, 0.2); e. (0.8, 0.2, 0.5); f. (0.8, 0.5, 0.2). The data in the table are the average accuracy of the 7 multimodal combinations.

[0099]

[0100] Please refer to Figure 4 , is the accuracy of two missing modes on the Audiovison-MNIST dataset: training uses all images and audio of different scales, but testing uses only images ( Figure 4 Left) or use both image and audio ( Figure 4 Right). The lower bound is LeNet5 trained only with images, and the upper bound is a multimodal model trained with images and audio. The comparison methods include AutoEncoder, GAN, ShaSpec, MBT-Sample, and the embodiment of the present invention (expressed using MCE). Figure 4 The horizontal axis is the audio usage rate, and the vertical axis is the accuracy rate.

[0101] Please refer to Figure 5-8 , which are the results of qualitative analysis of the four data sets.

[0102] According to the results of quantitative and qualitative analysis, the embodiments of the present invention have a low increase in time cost for all tasks in multiple data sets and can stably maintain their advantages in performance, indicating that they have significant effectiveness, superiority, universality, and versatility in learning, analyzing, and processing incomplete and unbalanced multimodal data.

[0103] Please refer to Figure 9, an embodiment of the present invention provides a multimodal data learning and analysis device, the device comprising:

[0104] An acquisition unit 900 is configured to acquire a multimodal dataset based on task requirements; the dataset includes a plurality of samples, each sample including at least some of all modalities;

[0105] a determining unit 902 for counting the number of occurrences of each modality in the samples of the data set, and determining a quotient of the total number of samples and the number of occurrences as a learning speed factor for each modality;

[0106] The determining unit 902 is further configured to calculate the learning capability upper limit and the current learning state of each modality, and determine the difference between the learning capability upper limit and the current learning state as the learning state factor of each modality;

[0107] The analysis unit 904 is used to take the multimodal data of the sample as input and the actual task results of the sample annotation as output, train the multimodal model, and use the loss function to update the model parameters of the multimodal model so as to use the trained multimodal model to learn and analyze the multimodal data; the loss function is constructed using the learning speed factor and learning state factor of each modality.

[0108] In one embodiment of the present invention, the upper limit of the learning ability of each modality is calculated as follows:

[0109] For each modality, the following steps are performed: A single-modality decoder is used to perform task-specific decoding on the single modality feature in the sample to predict the output task result. The accuracy is then calculated using the predicted output task result and the actual task result annotated with the sample; this accuracy is then determined as the upper limit of the learning ability of the modality.

[0110] The current learning state of each modality is calculated as follows:

[0111] For each modality, the following steps are performed: freely combining the other modalities in the data set except the modality to obtain multiple first subsets; for each first subset, calculating the contribution value that can be provided to the first subset after adding the modality to the first subset, and determining the sum of the contribution values ​​of the multiple first subsets as the current learning state of the single modality.

[0112] In one embodiment of the present invention, the calculating of the contribution value provided to the first subset after the mode is added to the first subset includes:

[0113]

[0114] in, To add mode m to the first subset The first subset The contribution value provided is, M is the number of modes contained in the dataset, The first subset The number of modes in b is the total number of samples, h m is the characteristic of mode m, f fusion (·) is the feature fusion network, f dec (·) is a multimodal decoder, is the actual task result of the sample annotation, f acc (·) is the function for calculating accuracy;

[0115] The sum of the contribution values ​​of multiple first subsets is calculated using the following formula:

[0116]

[0117] Among them, φ m is the total contribution value, and M1 is the number of the first subset.

[0118] In one embodiment of the present invention, the loss function is:

[0119]

[0120] in, is the total loss, The task result y predicted by the multimodal model and the actual task result of the sample annotation The first contrast loss between To achieve; Input each single modality feature to the single modality decoder separately The task results of the post-prediction output and the actual task results of the sample annotation The second contrast loss between is the learning speed factor of mode m, is the learning state factor of mode m; The third contrast loss is calculated as follows: after freely combining all the modes in the data set, multiple second subsets are obtained, and the second subsets are The modal features within are input into the feature fusion network for feature fusion, and the loss between the task result predicted and the actual task result of the sample annotation after the fused features are input into the multimodal decoder is calculated. The average of the sum of the losses corresponding to all second subsets is determined as the loss corresponding to a single sample, and the average of the losses corresponding to all samples is calculated, and the average is determined as the third comparative loss; M2 is the number of second subsets.

[0121] In one embodiment of the present invention, the feature fusion network includes a feature reconstruction network and an information fusion network;

[0122] The multimodal data of the sample is used as input and the actual task results of the sample annotation are used as output to train the multimodal model, including: inputting the multimodal data of the sample into the encoder of the corresponding modality to use the encoder to extract features, and inputting the extracted modal features into the feature reconstruction network so that the feature reconstruction network reconstructs and restores the modal features not included, outputting the modal features of all modalities to the information fusion network, and using the information fusion network to perform multimodal feature fusion to obtain multimodal fusion features.

[0123] In one embodiment of the present invention, the feature reconstruction network reconstructs and restores the modal features not included, including: selecting some modal features from the modal features included in the sample and discarding them; reconstructing and restoring the modal features not included in the sample and the discarded modal features using the modal features included in the sample and the modal features of the remaining modalities in the sample to output the modal features of all modalities; constructing a fourth contrast loss using the discarded modal features and the modal features reconstructed and restored for the discarded modalities, and updating the fourth contrast loss to the loss function to update the network parameters of the feature reconstruction network using the updated loss function.

[0124] It should be noted that the multimodal data learning and analysis device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the multimodal data learning and analysis device provided in the above embodiment and the multimodal data learning and analysis method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0125] The embodiment of the present application also provides a computer device, please refer to Figure 10 The computer device includes a processor and a memory, in which at least one instruction, at least one program, code set or instruction set is stored. The at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the multimodal data learning and analysis method provided by the above-mentioned method embodiments.

[0126] An embodiment of the present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the multimodal data learning and analysis method provided by the above-mentioned method embodiments.

[0127] An embodiment of the present application also provides a computer program product, which includes a computer program. The processor of a computer device reads the computer program from a computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the multimodal data learning and analysis method described in any of the above embodiments.

[0128] For the convenience of description, the above systems or devices are described as being divided into various modules or units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0129] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application or certain parts of the embodiments.

[0130] Finally, it should be noted that, in this document, relational terms such as first, second, third, and fourth are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0131] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A multimodal data learning and analysis method, characterized in that: The method comprises: Acquire a multimodal dataset based on task requirements; the dataset includes multiple samples, each sample including at least some of all modalities; Counting the number of occurrences of each modality in the samples of the data set, and determining the quotient of the total number of samples and the number of occurrences as the learning speed factor of each modality; Calculate the learning ability upper limit and current learning state of each modality, and determine the difference between the learning ability upper limit and the current learning state as the learning state factor of each modality; The multimodal data of the sample is used as input, and the actual task results of the sample annotation are used as output. The multimodal model is trained, and the model parameters of the multimodal model are updated using the loss function, so that the trained multimodal model can be used to learn and analyze the multimodal data; the loss function is constructed using the learning speed factor and learning state factor of each modality.

2. The method according to claim 1, characterized in that The upper limit of learning ability of each modality is calculated as follows: For each modality, the following steps are performed: A single-modality decoder is used to perform task-specific decoding on the single modality feature in the sample to predict the output task result. The accuracy is then calculated using the predicted output task result and the actual task result annotated with the sample; this accuracy is then determined as the upper limit of the learning ability of the modality. The current learning state of each modality is calculated as follows: For each modality, the following steps are performed: freely combining the other modalities in the data set except the modality to obtain multiple first subsets; for each first subset, calculating the contribution value that can be provided to the first subset after adding the modality to the first subset, and determining the sum of the contribution values ​​of the multiple first subsets as the current learning state of the single modality.

3. The method according to claim 2, characterized in that The calculating of the contribution value provided to the first subset after the mode is added to the first subset includes: in, To add mode m to the first subset The first subset The contribution value provided is, M is the number of modes contained in the dataset, The first subset The profit function, The first subset The number of modes in , N b is the total number of samples, h m is the characteristic of mode m, f fusion (·) is the feature fusion network, f dec (·) is a multimodal decoder, is the actual task result of the sample annotation, f acc (·) is the function for calculating accuracy; The sum of the contribution values ​​of multiple first subsets is calculated using the following formula: Among them, φ m is the total contribution value, and M1 is the number of the first subset.

4. The method according to claim 3, characterized in that The loss function is: in, is the total loss, The task result y predicted by the multimodal model and the actual task result of the sample annotation The first contrast loss between To achieve; Input each single modality feature to the single modality decoder separately The task results of the predicted output and the actual task results of the sample annotation The second contrast loss between is the learning speed factor of mode m, is the learning state factor of mode m; The third contrast loss is calculated as follows: after freely combining all the modes in the data set, multiple second subsets are obtained, and the second subsets are The modal features within are input into the feature fusion network for feature fusion, and the loss between the task result predicted and the actual task result of the sample annotation after the fused features are input into the multimodal decoder is calculated. The average of the sum of the losses corresponding to all second subsets is determined as the loss corresponding to a single sample, and the average of the losses corresponding to all samples is calculated, and the average is determined as the third comparative loss; M2 is the number of second subsets.

5. The method according to claim 2, characterized in that The feature fusion network includes a feature reconstruction network and an information fusion network; The multimodal data of the sample is used as input, and the actual task results of the sample annotation are used as output to train the multimodal model, including: The multimodal data of the sample is input into the encoder of each corresponding modality to use the encoder to extract features, and the extracted modal features are input into the feature reconstruction network to enable the feature reconstruction network to reconstruct and restore the modal features that are not included, and output the modal features of all modalities to the information fusion network to use the information fusion network to perform multimodal feature fusion and obtain multimodal fusion features.

6. The method according to claim 5, characterized in that The feature reconstruction network reconstructs and restores the modal features not included, including: Selecting and discarding some modal features from the modal features included in the sample; Reconstructing and restoring modal features not included in the sample and discarded modal features using the modal features included in the sample and the modal features of the remaining modalities in the sample, so as to output modal features of all modalities; A fourth contrast loss is constructed using the discarded modal features and the modal features reconstructed and restored for the discarded modality, and the fourth contrast loss is updated to the loss function to update the network parameters of the feature reconstruction network using the updated loss function.

7. A multimodal data learning and analysis device, characterized in that: The device comprises: An acquisition unit, configured to acquire a multimodal dataset based on task requirements; the dataset includes a plurality of samples, each sample including at least some of all modalities; a determination unit, configured to count the number of occurrences of each modality in the samples of the data set, and determine the quotient of the total number of samples and the number of occurrences as a learning speed factor for each modality; The determining unit is further configured to calculate the learning ability upper limit and the current learning state of each modality, and determine the difference between the learning ability upper limit and the current learning state as the learning state factor of each modality; An analysis unit is used to take the multimodal data of the sample as input and the actual task results of the sample annotation as output, train the multimodal model, and update the model parameters of the multimodal model using a loss function, so as to use the trained multimodal model to learn and analyze the multimodal data; the loss function is constructed using the learning speed factor and learning state factor of each modality.

8. A computer device, characterized in that: The computer device includes a memory and a processor, the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to implement the steps of any one of the methods described in claims 1-6.

9. A computer-readable storage medium, characterized in that The storage medium stores a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images