A method for optimizing multimodal feature learning efficiency based on low-rank factor decomposition
Through the multimodal feature learning method based on low rank factor decomposition, the problem of insufficient alignment of multimodal data is solved, the full integration of information and efficiency is achieved, and the learning efficiency and recognition accuracy of multimodal features are significantly improved. .
Patent Information
- Application Number
- CN202410623819.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-20
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-05-20
AI Technical Summary
The failure to effectively align multimodal data in the existing multimodal feature learning process leads to insufficient information integration, affecting the learning effect and accurate identification, and due to the complex model structure, many parameters and large computing resource requirements, the efficiency is low.
Using a multimodal feature learning method based on low rank factor decomposition, multimodal interaction relationship is constructed through hierarchical knowledge integration and approximate tensor fusion, low rank factor decomposition module is used to decompose the context vector, combined with 1D time convolution layer and cross-mode attention module to improve the learning efficiency of multimodal features.
Effectively aligning information of different modalities improves the learning efficiency and recognition accuracy of multimodal features, significantly enhances the model's processing ability of multiple data types, and overcomes the heterogeneous gap between multimodal data.
Smart Images

Figure CN118568658B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of multimodal feature learning efficiency optimization, and specifically relates to a multimodal feature learning efficiency optimization method based on low-rank factor decomposition. Background Art
[0002] Multimodal features refer to information features from different data sources or different types of sensors that capture different aspects of the same object or event. They are mainly used in machine learning and artificial intelligence.
[0003] However, the existing multimodal features fail to effectively align the knowledge of multimodal data during the learning process. Improper alignment will lead to insufficient information integration, which will in turn affect the learning effect and recognition accuracy. In addition, due to the high complexity of the learning model structure, the large number of parameters, and the large demand for computing resources, the efficiency is low when learning multimodal data.
[0004] In view of this, the present invention is proposed. Summary of the invention
[0005] The technical problem to be solved by the present invention is to overcome the deficiencies of the prior art and provide a method for optimizing the learning efficiency of multimodal features based on low-rank factor decomposition, thereby solving the problems raised in the above-mentioned background technology.
[0006] In order to solve the above technical problems, the basic concept of the technical solution adopted by the present invention is:
[0007] A method for optimizing multimodal feature learning efficiency based on low-rank factor decomposition, comprising the following steps:
[0008] Step 1: Receive and obtain various types of input data, and divide the collected input data into three different modes;
[0009] Step 2: Establish a backbone network and use the hierarchical knowledge integration method to learn the association of multimodal semantics;
[0010] Step 3: Use the approximate tensor fusion method to construct the interaction relationship between all single modalities, dual modalities, and tri-modalities, and use fewer backbone networks to obtain multimodal knowledge representation;
[0011] Step 4: Obtain the obtained modality and use the corresponding algorithm to generate the context vector of the corresponding modality data according to the type of modality;
[0012] Step 5: Set up a low-rank factor decomposition module to decompose the obtained context vector, and fuse the decomposed context vector into a multimodal fusion expression;
[0013] Step 6: The multimodal fusion expression passes through a 1D temporal convolution layer to obtain the local information of the modal fusion sequence;
[0014] Step 7: Use the Transformer operation that integrates cross-modal information, based on the cross-modal attention module, to enable one modality to receive information from another modality.
[0015] Optionally, the three different modalities are divided into text, audio, and video, respectively, and the feature semantic knowledge integration includes multimodal vector nodes, multimodal semantic nodes, knowledge base nodes, multimodal knowledge nodes, integration relationships between semantic knowledge, and real-time data nodes to be tested.
[0016] Optionally, the backbone network is a Multiway Transformer consisting of a self-attention knowledge base, a self-attention module, and multiple modal semantic modules, and the self-attention knowledge base includes visual knowledge, speech knowledge, and text knowledge.
[0017] Optionally, the steps of establishing a backbone network in Step 2 and using a hierarchical knowledge integration method to perform association learning on multimodal semantics are:
[0018] Step 2.1.1: The information obtained from the self-attention module and the modal semantic module will be fed into a feedforward neural network;
[0019] Step 2.1.2: Use the self-attention module to align the information from different modalities and deeply fuse and encode the information of different modalities so that it can be better applied to multimodal visual understanding tasks;
[0020] Step 2.1.3: According to the modality category of the current input, Multiway Transformer selects different knowledge modules to encode it to learn more modality-specific information.
[0021] Optionally, the steps of obtaining the modality in Step 4 and generating a context vector of the corresponding modality data using a corresponding algorithm according to the type of the modality are:
[0022] Step 4.1: The input video sequence will be enhanced by the dynamic particle data enhancement method, that is, the visual context vector containing context information is obtained;
[0023] Step 4.2: The audio sequence is subjected to a one-step LSTM data compression for data enhancement, that is, the audio context vector is obtained;
[0024] Step 4.3: The text sequence passes through the standard BERT language processing model to obtain the text context vector.
[0025] Optionally, the steps of the dynamic particle data enhancement method in Step 4.1 are:
[0026] Step 4.1.1: Rasterize the collected video data;
[0027] Step 4.1.2: Use a convolutional neural network (CNN) to process the rasterized image, then select the extracted edge points as the particle set, and set the corresponding initial values for the target representation model and the particle motion model;
[0028] Step 4.1.3: Iteration process of particles using the maximum likelihood estimation method (Estimate Maximum, EM);
[0029] Step 4.1.4: Determine the weight of each particle according to the Bhattacharyya coefficient;
[0030] Step 4.1.5: Adaptively adjust the parameters of the particle motion model according to the observed values of the target representation state;
[0031] Step 4.1.6: Obtain an estimated value for the representation of the video data, that is, complete the data enhancement of the video sequence.
[0032] Optionally, the multimodal fusion expression is passed through a 1D temporal convolution layer to obtain the local information of the modal fusion sequence as follows:
[0033] Step 6.1: Add position embedding to the temporal information representation, calculate the index information of each position embedding, and obtain the feature vector containing the low-level position information;
[0034] Step 6.2: Embed the feature vectors into the video sequence, audio sequence, and text sequence respectively, so that the video sequence, audio sequence, and text sequence contain time information.
[0035] Optionally, the low-rank factor decomposition module may be decomposed into a low-rank video factor, a low-rank audio factor, and a low-rank text factor.
[0036] Optionally, each cross-modal Transformer is composed of D layers of cross-modal attention modules, and cross-modal interactions are established between each pair of modalities to form a fused cross-modal Transformer.
[0037] Optionally, Step 2 also includes the following steps before establishing the backbone network to use the hierarchical knowledge integration method to learn the association of multimodal semantics:
[0038] Step 2.2.1: Design an association learning function to help the model learn and understand the relationship between different texts, images and videos, so that it has the ability of self-reasoning and can recognize unlearned knowledge;
[0039] Step 2.2.2: Project text, images and videos into the same dimension and establish the relationship between each dimension and the prediction result;
[0040] Step 2.2.3: Project in the order of 1D-2D-3D. The final loss is the average of the cross entropy losses of the three. The expression of the average of the cross entropy losses of the three is:
[0041]
[0042] In the formula, t is the text feature, i is the image feature, v is the video feature, r is the prediction box, and R n is the feature space, k is the feature dimension, τ is the adjustable temperature parameter, T is the rotation amount, and j∈R n .
[0043] After adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art. Of course, any product implementing the present invention does not necessarily need to achieve all the advantages described below at the same time:
[0044] This application adopts a knowledge integration method based on multimodal feature semantics to perform association learning on multimodal data. Knowledge alignment is performed by integrating a self-attention knowledge base and multiple knowledge modules. The shared self-attention module can effectively learn the knowledge alignment of different modalities and deeply fuse and encode different modal information, so that it can be better applied to multimodal behavior recognition tasks and improve the learning efficiency of multimodal features.
[0045] The specific implementation modes of the present invention are further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] The drawings described below are only some embodiments. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:
[0047] Figure 1 It is a block diagram of the low-rank factorization multimodal data fusion method;
[0048] Figure 2 Framework for multimodal semantic association learning for hierarchical knowledge integration;
[0049] It should be noted that these drawings and textual descriptions are not intended to limit the conceptual scope of the present invention in any way, but are intended to illustrate the concept of the present invention for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0050] The present invention will now be described in further detail with reference to the accompanying drawings.
[0051] See also Figure 1-2 As shown, in this embodiment, a method for optimizing multimodal feature learning efficiency based on low-rank factor decomposition is provided, comprising the following steps:
[0052] Step 1: Receive and obtain various types of input data, and divide the collected input data into three different modes;
[0053] Step 2: Establish a backbone network and use the hierarchical knowledge integration method to learn the association of multimodal semantics;
[0054] Step 2.1.1: The information obtained from the self-attention module and the modal semantic module will be fed into a feedforward neural network;
[0055] Step 2.1.2: Use the self-attention module to align the information from different modalities and deeply fuse and encode the information of different modalities so that it can be better applied to multimodal visual understanding tasks;
[0056] Step 2.1.3: According to the modality category of the current input, Multiway Transformer selects different knowledge modules to encode it to learn more modality-specific information.
[0057] It should be noted that the shared self-attention module can effectively learn the knowledge alignment of different modalities and deeply fuse and encode the information of different modalities, so that it can be better applied to multimodal behavior recognition tasks and improve the learning efficiency of multimodal features.
[0058] Specifically, by combining the self-attention module and the multimodal semantic module, it can effectively align and deeply fuse information of different modalities, thereby improving the performance of multimodal visual understanding tasks, and enabling Multiway Transformer to flexibly select appropriate knowledge modules for learning according to the input modality type, significantly enhancing the model's ability to process multiple data types.
[0059] Step 2 also includes the following steps before building the backbone network and using the hierarchical knowledge integration method to learn the association of multimodal semantics
[0060] Step 2.2.1: Design an association learning function to help the model learn and understand the relationship between different texts, images and videos, so that it has the ability of self-reasoning and can recognize unlearned knowledge;
[0061] Step 2.2.2: Project text, images and videos into the same dimension and establish the relationship between each dimension and the prediction result;
[0062] Step 2.2.3: Project in the order of 1D-2D-3D. The final loss is the average of the cross entropy losses of the three. The expression of the average of the cross entropy losses of the three is:
[0063]
[0064] In the formula, t is the text feature, i is the image feature, v is the video feature, r is the prediction box, and R n is the feature space, k is the feature dimension, τ is the adjustable temperature parameter, T is the rotation amount, and j∈R n .
[0065] Step 3: Use the approximate tensor fusion method to construct the interaction relationship between all single modalities, dual modalities, and tri-modalities, and use fewer backbone networks to obtain multimodal knowledge representation;
[0066] Step 4: Obtain the obtained modality and use the corresponding algorithm to generate the context vector of the corresponding modality data according to the type of modality;
[0067] Step 4.1: The input video sequence will be enhanced by the dynamic particle data enhancement method, that is, the visual context vector containing context information is obtained;
[0068] The steps of the dynamic particle data enhancement method in Step 4.1 are:
[0069] Step 4.1.1: Rasterize the collected video data;
[0070] Step 4.1.2: Use a convolutional neural network (CNN) to process the rasterized image, then select the extracted edge points as the particle set, and set the corresponding initial values for the target representation model and the particle motion model;
[0071] Step 4.1.3: Iteration process of particles using the maximum likelihood estimation method (Estimate Maximum, EM);
[0072] Step 4.1.4: Determine the weight of each particle according to the Bhattacharyya coefficient;
[0073] Step 4.1.5: Adaptively adjust the parameters of the particle motion model according to the observed values of the target representation state;
[0074] Step 4.1.6: Obtain an estimated value for the representation of the video data, that is, complete the data enhancement of the video sequence.
[0075] Step 4.2: The audio sequence is subjected to a one-step LSTM data compression for data enhancement, that is, the audio context vector is obtained;
[0076] Step 4.3: The text sequence passes through the standard BERT language processing model to obtain the text context vector.
[0077] It should be noted that the feature recognition accuracy of video data and the accuracy of target tracking are improved through rasterization and convolutional neural network processing combined with maximum likelihood estimation and Bhattacharyya coefficient optimization.
[0078] Step 5: Set up a low-rank factor decomposition module to decompose the obtained context vector, and fuse the decomposed context vector into a multimodal fusion expression;
[0079] It should be noted that after the three modal vectors are low-ranked, the heterogeneous gap between multimodal data is overcome.
[0080] Step 6: The multimodal fusion expression passes through a 1D temporal convolution layer to obtain the local information of the modal fusion sequence;
[0081] In Step 6, the multimodal fusion expression passes through a 1D temporal convolution layer to obtain the local information of the modal fusion sequence:
[0082] Step 6.1: Add position embedding to the temporal information representation, calculate the index information of each position embedding, and obtain the feature vector containing the low-level position information;
[0083] Step 6.2: Embed the feature vectors into the video sequence, audio sequence, and text sequence respectively, so that the video sequence, audio sequence, and text sequence contain time information.
[0084] Step 7: Use the Transformer operation that integrates cross-modal information, based on the cross-modal attention module, to enable one modality to receive information from another modality.
[0085] It should be noted that the method proposed in this application can directly process unaligned sequences and use a learning method to imitate the collaborative ability of humans to express multimodal data.
[0086] In this embodiment, the three different modalities are divided into text, audio, and video, respectively, and the feature semantic knowledge integration includes multimodal vector nodes, multimodal semantic nodes, knowledge base nodes, multimodal knowledge nodes, integration relationships between semantic knowledge, and real-time data nodes to be tested.
[0087] In this embodiment, the backbone network is a Multiway Transformer composed of a self-attention knowledge base, a self-attention module and semantic modules of multiple modalities, and the self-attention knowledge base contains visual knowledge, speech knowledge and text knowledge.
[0088] In this embodiment, the multimodal fusion expression passes through a 1D temporal convolution layer to obtain the local information of the modal fusion sequence in the following steps:
[0089] We add position embedding to the temporal information representation and calculate the index information of each position embedding to obtain the feature vector containing the low-level position information.
[0090] The feature vectors are embedded in the video sequence, audio sequence and text sequence respectively, so that the video sequence, audio sequence and text sequence contain time information.
[0091] It should be noted that by adding position embedding and calculating the index information of each position embedding, it can be ensured that the sequence can contain time information.
[0092] In this embodiment, the low-rank factor decomposition module is decomposed into a low-rank video factor, a low-rank audio factor, and a low-rank text factor.
[0093] In this embodiment, each cross-modal Transformer is composed of D layers of cross-modal attention modules, and cross-modal interaction is established between each pair of modalities to form a fused cross-modal Transformer.
[0094] The present invention is not limited to the above-mentioned embodiments. Anyone should be aware that any structural changes made under the enlightenment of the present invention, and any technical solutions that are the same or similar to the present invention, fall within the protection scope of the present invention. The technology, shape, and structural parts not described in detail in the present invention are all well-known technologies.
Claims
1. A method for optimizing multimodal feature learning efficiency based on low-rank factor decomposition, characterized in that: The following steps are involved: Step 1: Receive and obtain various types of input data, and divide the collected input data into three different modes; Step 2: Establish a backbone network and use the hierarchical knowledge integration method to learn the association of multimodal semantics; Step 3: Use the approximate tensor fusion method to construct the interaction relationship between all single modalities, dual modalities, and tri-modalities, and reduce the number of independent backbone networks used to obtain multimodal knowledge representation; Step 4: Obtain the obtained modality and use the corresponding algorithm to generate the context vector of the corresponding modality data according to the type of modality; Step 5: Set up a low-rank factor decomposition module to decompose the obtained context vector, and fuse the decomposed context vector into a multimodal fusion expression; Step 6: The multimodal fusion expression passes through a 1D temporal convolution layer to obtain the local information of the modal fusion sequence; Step 7: Use the Transformer operation that integrates cross-modal information, based on the cross-modal attention module, to enable one modality to receive information from another modality; The three different modalities are divided into text, audio, and video. The feature semantic knowledge integration includes multimodal vector nodes, multimodal semantic nodes, knowledge base nodes, multimodal knowledge nodes, integration relationships between semantic knowledge, and real-time data nodes to be tested. The backbone network is a Multiway Transformer consisting of a self-attention knowledge base, a self-attention module, and a multi-modal semantic module. The self-attention knowledge base contains visual knowledge, speech knowledge, and text knowledge. In Step 2, the steps of building a backbone network and using a hierarchical knowledge integration method to learn multimodal semantic associations are as follows: Step 2.1.1: The information obtained from the self-attention module and the modal semantic module will be fed into a feedforward neural network; Step 2.1.2: Use the self-attention module to align the information from different modalities and deeply fuse and encode the information from different modalities to improve the performance of the model in multimodal visual understanding tasks; Step 2.1.3: According to the current input modality category, Multiway Transformer selects different knowledge modules to encode it to extract rich modality-specific information; The steps of obtaining the modality obtained in Step 4 and using the corresponding algorithm to generate the context vector of the corresponding modality data according to the type of modality are as follows: Step 4.1: The input video sequence will be enhanced by the dynamic particle data enhancement method, that is, the visual context vector containing context information is obtained; Step 4.2: The audio sequence is subjected to a one-step LSTM data compression for data enhancement, that is, the audio context vector is obtained; Step 4.3: The text sequence passes through the standard BERT language processing model to obtain the text context vector.
2. According to claim 1, a multimodal feature learning efficiency optimization method based on low-rank factor decomposition is characterized in that: Step 2 also includes the following steps before building the backbone network and using the hierarchical knowledge integration method to learn the association of multimodal semantics Step 2.2.1: Design an association learning function to help the model learn and understand the relationship between different texts, images and videos, so that it has the ability of self-reasoning and can recognize unlearned knowledge; Step 2.2.2: Project text, images and videos into the same dimension and establish the relationship between each dimension and the prediction result; Step 2.2.3: Project in the order of 1D-2D-3D. The final loss is the average of the cross entropy losses of the three. The expression of the average of the cross entropy losses of the three is: In the formula, t is the text feature, i is the image feature, v is the video feature, r is the prediction box, and R n is the feature space, k is the feature dimension, τ is the adjustable temperature parameter, T is the rotation amount, and j∈R n .
3. The method for optimizing multimodal feature learning efficiency based on low-rank factor decomposition according to claim 2, characterized in that: The steps of the dynamic particle data enhancement method in Step 4.1 are: Step 4.1.1: Rasterize the collected video data; Step 4.1.2: Use a convolutional neural network (CNN) to process the rasterized image, then select the extracted edge points as the particle set, and set the corresponding initial values for the target representation model and the particle motion model; Step 4.1.3: Iteration process of particles using the maximum likelihood estimation method (Estimate Maximum, EM); Step 4.1.4: Determine the weight of each particle according to the Bhattacharyya coefficient; Step 4.1.5: Adaptively adjust the parameters of the particle motion model according to the observed values of the target representation state; Step 4.1.6: Obtain an estimated value for the representation of the video data, that is, complete the data enhancement of the video sequence.
4. The method for optimizing multimodal feature learning efficiency based on low-rank factor decomposition according to claim 1, characterized in that: In Step 6, the multimodal fusion expression passes through a 1D temporal convolution layer to obtain the local information of the modal fusion sequence: Step 6.1: Add position embedding to the temporal information representation, calculate the index information of each position embedding, and obtain the feature vector containing the low-level position information; Step 6.2: Embed the feature vectors into the video sequence, audio sequence, and text sequence respectively, so that the video sequence, audio sequence, and text sequence contain time information.
5. The method for optimizing multimodal feature learning efficiency based on low-rank factor decomposition according to claim 1, characterized in that: After decomposition, the low-rank factor decomposition module is divided into low-rank video factor, low-rank audio factor, and low-rank text factor.
6. The method for optimizing multimodal feature learning efficiency based on low-rank factor decomposition according to claim 1, characterized in that: Each cross-modal Transformer consists of D layers of cross-modal attention modules, and cross-modal interactions are established between each pair of modalities to form a fused cross-modal Transformer.
Citation Information
Patent Citations
Establishment method and system of multi-modal knowledge representation automatic-learning model
CN107480194A
Multi-modal knowledge intrinsic representation learning method and device, equipment and storage medium
CN114610911A
Aesthetics quality evaluation model and method based on multi-modal learning
CN115601772A
Transform-based multi-modal knowledge graph dynamic adaptive alignment method
CN116932777A
Multi-modal knowledge graph representation learning method based on cross-modal semantic alignment
CN117435744A
