Large model training optimization method and device based on multi-modal data fusion
By extracting local and global features from multimodal data and employing a hierarchical attention mechanism and dynamic loss function, the problems of insufficient semantic alignment and limitations of static loss functions in multimodal data fusion are solved, thereby improving the semantic consistency and generalization ability of the model and achieving efficient and accurate multimodal data processing.
Patent Information
- Application Number
- CN202511288511.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-10
- Publication Date
- 2026-01-13
AI Technical Summary
Multimodal data fusion suffers from problems such as insufficient semantic alignment, limitations of static loss functions, and an imbalance between computational efficiency and model performance, which limits the model's generalization ability.
By extracting local and global features from multimodal data, a hierarchical attention mechanism and a dynamic loss function are used to achieve cross-modal alignment and training optimization, and the weights are dynamically adjusted to improve semantic consistency and generalization ability.
It improves multimodal semantic consistency and the model's ability to discriminate different scenarios, enhances the model's generalization ability and training efficiency, and achieves efficient and accurate multimodal data processing.
Smart Images
Figure CN121328641A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence technology and multimodal large model technology. Specifically, this invention relates to a method and apparatus for training and optimizing large models based on multimodal data fusion. Background Technology
[0002] With the rapid development of artificial intelligence technology, Vision-Language Multi-modal Models (VLMMs) have demonstrated powerful capabilities in tasks such as image description generation, visual question answering, and cross-modal retrieval.
[0003] The following challenges remain in the process of multimodal data fusion and model training: The lack of semantic alignment in multimodal data, and the significant differences between visual and linguistic data in statistical characteristics and semantic expression levels, make it difficult to effectively integrate information between modalities.
[0004] The limitations of static loss functions: Traditional loss functions assign fixed weights during training, making it difficult to adapt to the dynamic needs of different training stages, thus limiting the model's generalization ability.
[0005] Balancing computational efficiency with model performance, insufficient collaborative optimization of high-dimensional feature extraction and low-dimensional representation during multimodal data fusion can easily lead to problems such as high computational complexity and slow training convergence. Summary of the Invention
[0006] The technical problem to be solved by the present invention is to provide a method and apparatus for training and optimizing large models based on multimodal data fusion, which aims to solve at least one of the above-mentioned technical problems.
[0007] Firstly, the technical solution of the present invention to solve the above-mentioned technical problems is as follows: a large model training optimization method based on multimodal data fusion, the method comprising: Acquire multimodal data and extract the initial features corresponding to each modality. For each modality, the initial features include global features and multiple local features. Based on the initial features corresponding to the data of all modalities, the local alignment features and global alignment features corresponding to the data of each modality are extracted. The local alignment features represent the semantically related local features in the data of different modalities, and the global alignment features represent the semantically related global features in the data of different modalities. Based on the local and global alignment features corresponding to the data of each modality, determine the aligned features corresponding to each modality; Based on the aligned features corresponding to all modalities, the initial model is trained to obtain a large multimodal model. The large multimodal model is then used to process the multimodal data to be processed, thereby obtaining the semantics corresponding to the multimodal data to be processed.
[0008] The beneficial effects of this invention are: by first aligning the local and global features of each modality across modalities, and then fusing the aligned features for training, visual, textual, and sensor information can be accurately corresponded in a unified semantic space. This not only improves the semantic consistency of multimodalities but also enhances the model's ability to discriminate differences in different scenarios. Ultimately, a large multimodal model with strong generalization ability and direct deployment is obtained, enabling efficient and accurate semantic understanding and reasoning of any multimodal data to be processed.
[0009] Based on the above technical solution, the present invention can be further improved as follows.
[0010] Furthermore, based on the initial features corresponding to the data of all modalities, the local alignment features corresponding to the data of each modality are extracted, including: Project all local features corresponding to the data of each modality onto the same dimension to obtain the first intermediate feature corresponding to each local feature; Determine the first semantic correlation between every two first intermediate features in the data of each modality; For each local feature in each modality, based on the first semantic relevance corresponding to the local feature, determine the local features corresponding to other modalities that are semantically related to the local feature, and based on the local features corresponding to other modalities that are semantically related to the local feature, determine the aligned local features corresponding to the local feature. For each modality, the local alignment features corresponding to the modality are obtained based on the aligned local features corresponding to all local features of the modality.
[0011] Furthermore, based on the initial features corresponding to the data of all modalities, the local alignment features corresponding to the data of each modality are extracted, including: Project all local features corresponding to the data of each modality onto the same dimension to obtain the first intermediate feature corresponding to each local feature; Determine the first semantic correlation between every two first intermediate features in the data of each modality; For each local feature in each modality, based on the first semantic relevance corresponding to the local feature, determine the local features corresponding to other modalities that are semantically related to the local feature, and based on the local features corresponding to other modalities that are semantically related to the local feature, determine the aligned local features corresponding to the local feature. For each modality, the local alignment features corresponding to the modality are obtained based on the aligned local features corresponding to all local features of the modality.
[0012] Furthermore, the method also includes: Based on the aligned features corresponding to each modality and the weights corresponding to each modality, the aligned features corresponding to all modalities are weighted and fused to obtain the fused features; The above-mentioned initial model is trained based on the aligned features corresponding to all modalities to obtain a large multimodal model, including: The initial model is trained based on the fusion features to obtain a multimodal large model.
[0013] Furthermore, the aforementioned multimodal data includes multimodal data corresponding to different scenarios. Multimodal data corresponding to the same scenario is taken as positive samples, and multimodal data corresponding to different scenarios is taken as negative samples. Then, the aligned features corresponding to all modalities include the aligned features corresponding to all positive samples and the aligned features corresponding to all negative samples. The above-mentioned initial model is trained based on the aligned features corresponding to all modalities to obtain a large multimodal model, including: Based on the aligned features corresponding to all positive samples and the aligned features corresponding to all negative samples, the contrastive learning loss value is determined. The contrastive learning loss value represents the difference between the aligned features corresponding to positive samples and the aligned features corresponding to negative samples. Based on the aligned features corresponding to all modalities, the cross-modal projection loss value is determined, which characterizes the semantic differences between the aligned features corresponding to different modalities. Based on the weights corresponding to each mode, a dynamic regularization term is determined. The dynamic regularization term characterizes the difference between the weights corresponding to each mode and the ideal weights. The initial model is trained using the contrastive learning loss, cross-modal projection loss, and dynamic regularization term to obtain a multimodal large model.
[0014] Secondly, to solve the above-mentioned technical problems, the present invention also provides a large model training and optimization device based on multimodal data fusion, the device comprising: The acquisition module is used to acquire multimodal data and extract the initial features corresponding to the data of each modality; The feature alignment module is used to extract the local alignment features and global alignment features corresponding to the data of each modality based on the initial features corresponding to the data of all modalities. The local alignment features represent the semantically related local features in the data of different modalities, and the global alignment features represent the semantically related global features in the data of different modalities. The fusion module is used to determine the aligned features for each modality based on the local and global alignment features corresponding to the data of each modality. The training module is used to train the initial model based on the aligned features corresponding to all modalities to obtain a large multimodal model. This large multimodal model is then used to process the multimodal data to be processed, thereby obtaining the semantics corresponding to the multimodal data to be processed.
[0015] Thirdly, in order to solve the above-mentioned technical problems, the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the large model training and optimization method based on multimodal data fusion of the present application.
[0016] Fourthly, in order to solve the above-mentioned technical problems, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the large model training and optimization method based on multimodal data fusion of the present application.
[0017] Additional aspects and advantages of this application will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of this application. Attached Figure Description
[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments of the present invention will be briefly introduced below.
[0019] Figure 1 A flowchart illustrating a large model training optimization method based on multimodal data fusion, provided as an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a large model training and optimization device based on multimodal data fusion, provided in one embodiment of the present invention. Figure 3 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present invention. Detailed Implementation
[0020] The principles and features of the present invention are described below. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.
[0021] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.
[0022] The solution provided in this invention can be applied to any application scenario that requires optimization training of large multimodal models. The solution provided in this invention can be executed by any electronic device, such as a user's terminal device, including at least one of the following: smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart TV, or smart in-vehicle device.
[0023] This invention provides a possible implementation, such as... Figure 1 The diagram shows a flowchart of a large model training optimization method based on multimodal data fusion. This method can be executed by any electronic device, such as a terminal device, or jointly executed by a terminal device and a server. For ease of description, the method provided in this embodiment will be described below using a terminal device as the execution subject as an example. Figure 1 The flowchart shown indicates that the method may include the following steps: S10, acquire multimodal data and extract the initial features corresponding to the data of each modality. For the data of each modality, the initial features corresponding to the modality include global features and multiple local features. S20. Based on the initial features corresponding to the data of all modalities, extract the local alignment features and global alignment features corresponding to the data of each modality. The local alignment features represent the semantically related local features in the data of different modalities, and the global alignment features represent the semantically related global features in the data of different modalities. S30, determine the aligned features for each modality based on the local and global alignment features corresponding to the data of each modality; S40. Based on the aligned features corresponding to all modalities, the initial model is trained to obtain a large multimodal model. The large multimodal model is then used to process the multimodal data to be processed, thereby obtaining the semantics corresponding to the multimodal data to be processed.
[0024] The method of this invention first aligns the local and global features of each modality across modalities, and then fuses the aligned features for training. This enables visual, textual, and sensor information to accurately correspond in a unified semantic space, improving multimodal semantic consistency and enhancing the model's ability to discriminate differences in different scenarios. Ultimately, it yields a large multimodal model with strong generalization ability that can be directly deployed, enabling efficient and accurate semantic understanding and reasoning of any multimodal data to be processed.
[0025] The following specific embodiments further illustrate the solution of the present invention. In this embodiment, a large model training optimization method combining multimodal data fusion and dynamic loss function is proposed. By dynamically adjusting the weights of the loss function and optimizing the cross-modal alignment strategy, the semantic consistency, generalization ability and training efficiency of the model are significantly improved.
[0026] Specifically, the large model training optimization method based on multimodal data fusion provided in this embodiment may include the following steps: S10, acquire multimodal data and extract the initial features corresponding to each modality. For each modality, the initial features corresponding to the modality include global features and multiple local features. Each local feature corresponds to a local data in the modality data (if it is an image, the local data can be a part of the image; if it is text, the local data can be a fragment of the text). Multimodal data includes image (or video) data, text data, and sensor data. Sensor data refers to any digitizable sensing information other than images and text acquired by sensors, including but not limited to: depth maps output by depth cameras, LiDAR point clouds, IMU three-axis acceleration / angular velocity, microphone audio waveforms, temperature, humidity or air pressure readings, millimeter-wave radar range-Doppler maps, and wearable heart rate and blood oxygen signals. All data has been converted from analog to digital and preprocessed to form discrete digital sampling sequences or tensors of the same dimension as visual and text data, which are used for alignment and fusion with image and text features.
[0027] In this context, the global feature corresponding to each modality's data refers to a single fixed-dimensional vector obtained by global pooling (such as average pooling, max pooling, or attention-weighted summation) of all local features (or original data) of that modality. It condenses the overall semantic information of that modality and is used for cross-modal global alignment with the global features of other modalities. The local features corresponding to each modality's data refer to the independently representable small unit feature vectors obtained by segmenting the data of that modality according to spatial or temporal dimensions. For example, patch features after image segmentation, token features after text word segmentation, and segment features after sensor sequence segmentation. All these local units together constitute the local feature set of that modality, which is used for subsequent cross-modal local alignment.
[0028] Optionally, before obtaining the initial features, the data for different modalities can be standardized separately to obtain the standardized features corresponding to each modality, which can be specifically represented as follows: in, , , These represent the standardized features corresponding to the visual, text, and sensor modalities, respectively.
[0029] S20. Based on the initial features corresponding to the data of all modalities, extract the local alignment features and global alignment features corresponding to the data of each modality. The local alignment features represent the semantically related local features in the data of different modalities, and the global alignment features represent the semantically related global features in the data of different modalities. One approach is to use hierarchical attention, which aligns features at two levels: local features (such as image regions and text words) and global features (such as the image as a whole and sentences).
[0030] Optionally, in S20 above, based on the initial features corresponding to the data of all modalities, the local alignment features corresponding to the data of each modality are extracted, including: S21, project all local features corresponding to the data of each modality onto the same dimension to obtain the first intermediate feature corresponding to each local feature; optionally, all local features of each modality (image patch, text token, sensor fragment) can be mapped to a unified fixed dimension d_target at once through the same or their respective corresponding learnable linear mapping matrix (shape d_in×d_target), so that the three types of local features are consistent in numerical space dimension, which is convenient for subsequent calculation of cross-modal semantic correlation.
[0031] S22, determine the first semantic correlation between every two first intermediate features in the data of each modality; wherein, calculate the dot product or inner product attention score between each pair of the three types of local features (visual patch, text token, sensor fragment) after unifying the dimension, and obtain the correlation score between each pair of local features of different modalities, which is used to determine which local regions, words or fragments correspond to each other semantically.
[0032] S23, for each local feature in each modality, based on the first semantic relevance corresponding to the local feature, determine the local features corresponding to other modalities that are semantically related to the local feature, and based on the local features corresponding to other modalities that are semantically related to the local feature, determine the aligned local features corresponding to the local feature; wherein, the local features corresponding to other modalities that are semantically related to the local feature are fused with the local feature, and the resulting aligned local features can learn the local features corresponding to other modalities, thereby improving the semantic expressive power of the local features and realizing cross-modal learning of local features.
[0033] S24, for each modality, obtain the local alignment features corresponding to the modality based on the aligned local features corresponding to all local features corresponding to the modality. Specifically, the aligned local features corresponding to all local features corresponding to the modality can be weighted and fused to obtain the local alignment features corresponding to that modality.
[0034] One approach to determining the aligned local feature for each local feature in each modality, based on local features in other modalities that are semantically related to the local feature, is as follows: Based on the local features corresponding to other modalities that are semantically related to the local features, a first alignment feature is determined between the local features and the local features corresponding to each modality in the other modalities that are semantically related to the local features. Then, based on all the first alignment features corresponding to the local features, the aligned local features corresponding to the local features are obtained.
[0035] Based on the same principle described above, in S20, global alignment features corresponding to the data of each modality are extracted according to the initial features corresponding to the data of all modalities, including: S25 projects the global features corresponding to all modal data onto the same dimension to obtain the second intermediate feature corresponding to each modality; the specific implementation method can be referred to in S21 above, and will not be repeated here.
[0036] S26, determine the second semantic relevance between the second intermediate features corresponding to each pair of modalities; specifically, for the second intermediate features of each modality after unified dimensional projection, calculate the cosine similarity (or inner product attention score) pairwise to obtain the global semantic matching strength between each pair of modalities, which is used as the second semantic relevance.
[0037] S27. For each modality, based on all second semantic relevances corresponding to the modality, determine the global features corresponding to other modalities that are semantically related to the global features corresponding to the modality. Then, based on the global features corresponding to other modalities that are semantically related to the global features corresponding to the modality, determine the global alignment features corresponding to the modality. Global alignment features reflect the consistency and correspondence of overall semantics across modalities, i.e., the degree of semantic matching among the visual, textual, and sensor modalities at the same global scene level. The global alignment features of one modality can learn the global features of the other two modalities across modalities, improving the semantic expressive power of the global features.
[0038] One approach to determining the global alignment features for each modality, based on the global features of other modalities that are semantically related to the global features of the modality, is as follows: Based on the global features corresponding to other modalities that are related to the global feature semantics, a second alignment feature is determined between the global feature and the global features related to the global feature semantics corresponding to each modality in other modalities. Then, based on all the second alignment features corresponding to the modality, the global alignment feature corresponding to the modality is obtained.
[0039] Optionally, in determining the first semantic relevance between any two first intermediate features, for a local feature of a modality, the semantic similarity between that local feature and other local features of other modalities can be measured based on local weights. Similarly, in determining the second semantic relevance between any two second intermediate features of a modality, for a modality, the semantic similarity between that global feature and other global features of other modalities can be measured based on global weights.
[0040] The local weights between any two local features are different, and the global weights between any two global features are also different; both are dynamically changing.
[0041] As an example, if the data from any two modalities are images and text, a local weight between the images and text can be represented as: in, Indicates local weights, This represents a first intermediate feature corresponding to the image. This represents the transpose of a first intermediate feature corresponding to the text.
[0042] A global weight between an image and text can be represented as: in, Represents the global weight. This represents a second intermediate feature. This represents the transpose of a second intermediate feature corresponding to the text.
[0043] S30, determine the aligned features for each modality based on the local and global alignment features corresponding to the data of each modality; Specifically, the local alignment features and global alignment features corresponding to the data of each modality can be fused or spliced to obtain the aligned features corresponding to each modality.
[0044] Optionally, prior to S40, the method further includes: Based on the aligned features corresponding to each modality and the weights corresponding to each modality, the aligned features corresponding to all modalities are weighted and fused to obtain the fused features; The weights for each modality can be dynamically allocated. During model training, the aligned features of each modality can be integrated through a learnable weight allocation mechanism (Dynamic Weighting).
[0045] Specifically, it can be expressed as: in, Indicates the fusion weight. These are dynamic weight parameters, meaning the weights corresponding to each modality, which are adaptively adjusted during the training process. , and These represent the aligned features corresponding to different modalities.
[0046] in, These represent the learnable weight vectors, which are optimized through backpropagation.
[0047] S40. Based on the aligned features corresponding to all modalities, the initial model is trained to obtain a large multimodal model. The large multimodal model is then used to process the multimodal data to be processed, thereby obtaining the semantics corresponding to the multimodal data to be processed.
[0048] After obtaining the fused features, in step S40 above, the initial model is trained based on the aligned features corresponding to all modalities to obtain a large multimodal model, including: The initial model is trained based on the fusion features to obtain a multimodal large model.
[0049] In this application's scheme, a dynamic loss function is set up to improve the model's adaptability and generalization ability at different training stages by dynamically adjusting the weight distribution of the loss function. Specifically, in S40 above, The aforementioned multimodal data includes multimodal data corresponding to different scenarios. Multimodal data corresponding to the same scenario are taken as positive samples, and multimodal data corresponding to different scenarios are taken as negative samples. Then, the aligned features corresponding to all modalities include the aligned features corresponding to all positive samples and the aligned features corresponding to all negative samples. In this context, "same scene" refers to a set of raw data collected simultaneously, in the same space, and with the same semantic context. Based on the aligned features corresponding to all modalities, the initial model is trained to obtain a large multimodal model, including: S41. Based on the aligned features corresponding to all positive samples and the aligned features corresponding to all negative samples, determine the contrastive learning loss value. The contrastive learning loss value represents the difference between the aligned features corresponding to positive samples and the aligned features corresponding to negative samples. By using the contrastive learning loss value, the distance between positive sample pairs can be shortened and the distance between negative sample pairs can be widened, enabling the model to learn a semantic space that can distinguish whether they belong to the same real scene.
[0050] S42, Based on the aligned features corresponding to all modalities, determine the cross-modal projection loss value, which characterizes the semantic differences between the aligned features corresponding to different modalities; S43, determine the dynamic regularization term based on the weights corresponding to each mode. The dynamic regularization term characterizes the difference between the weights corresponding to each mode and the ideal weights. S44. The initial model is trained based on the contrastive learning loss value, the cross-modal projection loss value, and the dynamic regularization term to obtain a multimodal large model.
[0051] The above-mentioned method of training the initial model based on the contrastive learning loss, cross-modal projection loss, and dynamic regularization term to obtain the multimodal large model is as follows: The total loss value is determined based on the contrastive learning loss value, the cross-modal projection loss value, and the dynamic regularization term. The initial model is then trained based on the total loss value to obtain a multimodal large model.
[0052] The total loss value can be expressed as: in, This represents the total loss value. This represents the contrastive learning loss value. This represents the cross-modal projection loss value. Represents dynamic regularization terms. These are dynamic weighting coefficients that are adaptively adjusted according to the training phase.
[0053] As an example, suppose the data for any two modalities are images and text, respectively; a positive sample consists of images and text from the same scene, and a negative sample consists of images and text from different scenes. Then, what is the contrastive learning loss between the positive and negative samples for these two modalities? It can be represented as: in, This represents the aligned features of the image in scene A. This represents the aligned features of the text in scenario A. This represents the aligned features of the text in scenario B. and To form a positive sample, and Form a negative sample. Here, N is the temperature parameter, and N is the number of negative samples.
[0054] The semantic differences between aligned features corresponding to different modalities can be represented as: in, and These are the learnable projection matrices.
[0055] The dynamic regularization term can be expressed as: in, For the first The dynamic decay coefficient of the layer is dynamically adjusted according to the weight gradient. This represents the weight corresponding to the i-th layer.
[0056] in, For hyperparameters, control the gradient pair The impact.
[0057] In this application, the first... A layer refers to the level at which the i-th learnable parameter (weight matrix) resides in the initial model. Specifically, it could be the i-th block of a Transformer, the i-th feedforward layer, or the i-th set of weight tensors in a convolutional / linear layer. In other words, each step in the above scheme can be understood as a corresponding layer. If weights are involved in this layer, they can be dynamically adjusted using the formula corresponding to the dynamic regularization term mentioned above.
[0058] In this application's solution, dynamic weight parameters are initialized during the model training phase. and loss function coefficients In each iteration, the contrastive learning loss, cross-modal projection loss, and dynamic regularization term are calculated and dynamically adjusted based on the current training stage. ,For example: Where T is the total number of training rounds, This is the gradient-aware function. Model optimization and deployment involve optimizing the joint loss function using Monte Carlo simulation to generate high-quality image sequences. A lightweight version is deployed on edge devices to enable real-time multimodal interaction.
[0059] Through the above technical solutions, this invention has achieved innovation in multimodal data fusion and dynamic loss function design, significantly improving the semantic consistency, training efficiency and generalization ability of large models, and has broad industrialization prospects.
[0060] The solution of the present invention has the following beneficial effects: 1. Existing multimodal alignment strategies employ linear fusion with fixed weights, lacking hierarchical alignment; the loss function is designed as a static loss function (such as cross-entropy), with fixed weights; training efficiency and performance suffer from slow convergence and limited generalization ability. 2. The multimodal alignment strategy in this scheme adopts a hierarchical attention mechanism (local + global) with dynamic weight allocation; the loss function is designed as a dynamic loss function (contrastive learning + projection loss + regularization), with weights adaptively adjusted during the training phase; in terms of training efficiency and performance, dynamic weight optimization accelerates convergence, improving semantic alignment accuracy by 15%-20%.
[0061] Based on and Figure 1 Based on the same principle as the method shown, this embodiment of the invention also provides a large model training optimization device 20 based on multimodal data fusion, such as... Figure 2 As shown, the large model training and optimization device 20 based on multimodal data fusion may include an acquisition module 210, a feature alignment module 220, a fusion module 230, and a training module 240, wherein: The acquisition module 210 is used to acquire multimodal data and extract the initial features corresponding to the data of each modality. For the data of each modality, the initial features corresponding to the modality include global features and multiple local features. The feature alignment module 220 is used to extract the local alignment features and global alignment features corresponding to the data of each modality based on the initial features corresponding to the data of all modalities. The local alignment features represent the semantically related local features in the data of different modalities, and the global alignment features represent the semantically related global features in the data of different modalities. The fusion module 230 is used to determine the aligned features corresponding to each modality based on the local alignment features and global alignment features corresponding to the data of each modality; The training module 240 is used to train the initial model based on the aligned features corresponding to all modalities to obtain a large multimodal model, and then to process the multimodal data to be processed based on the large multimodal model to obtain the semantics corresponding to the multimodal data to be processed.
[0062] Optionally, when the feature alignment module 220 extracts the local alignment features corresponding to the data of each modality based on the initial features corresponding to the data of all modalities, it is specifically used for: Project all local features corresponding to the data of each modality onto the same dimension to obtain the first intermediate feature corresponding to each local feature; Determine the first semantic correlation between every two first intermediate features in the data of each modality; For each local feature in each modality, based on the first semantic relevance corresponding to the local feature, determine the local features corresponding to other modalities that are semantically related to the local feature, and based on the local features corresponding to other modalities that are semantically related to the local feature, determine the aligned local features corresponding to the local feature. For each modality, the local alignment features corresponding to the modality are obtained based on the aligned local features corresponding to all local features of the modality.
[0063] Optionally, when the feature alignment module 220 extracts the global alignment features corresponding to the data of each modality based on the initial features corresponding to the data of all modalities, it is specifically used for: Project the global features corresponding to all modal data onto the same dimension to obtain the second intermediate feature corresponding to each modality; Determine the second semantic relevance between the second intermediate features corresponding to each pair of modalities; For each modality, based on all the second semantic relevances corresponding to the modality, determine the global features corresponding to other modalities that are semantically related to the global features corresponding to the modality, and based on the global features corresponding to other modalities that are semantically related to the global features corresponding to the modality, determine the global alignment features corresponding to the modality.
[0064] Optionally, the device further includes: The fusion feature determination module is used to perform weighted fusion of the aligned features corresponding to all modes based on the aligned features corresponding to each mode and the weights corresponding to each mode, so as to obtain the fusion features; When the training module 240 trains the initial model based on the aligned features corresponding to all modalities to obtain a large multimodal model, it is specifically used for: The initial model is trained based on the fusion features to obtain a multimodal large model.
[0065] Optionally, the multimodal data mentioned above includes multimodal data corresponding to different scenarios. Multimodal data corresponding to the same scenario is taken as positive samples, and multimodal data corresponding to different scenarios is taken as negative samples. Then, the aligned features corresponding to all modalities include the aligned features corresponding to all positive samples and the aligned features corresponding to all negative samples. When the training module 240 trains the initial model based on the aligned features corresponding to all modalities to obtain a large multimodal model, it is specifically used for: Based on the aligned features corresponding to all positive samples and the aligned features corresponding to all negative samples, the contrastive learning loss value is determined. The contrastive learning loss value represents the difference between the aligned features corresponding to positive samples and the aligned features corresponding to negative samples. Based on the aligned features corresponding to all modalities, the cross-modal projection loss value is determined, which characterizes the semantic differences between the aligned features corresponding to different modalities. Based on the weights corresponding to each mode, a dynamic regularization term is determined. The dynamic regularization term characterizes the difference between the weights corresponding to each mode and the ideal weights. The initial model is trained using the contrastive learning loss, cross-modal projection loss, and dynamic regularization term to obtain a multimodal large model.
[0066] The large model training and optimization device based on multimodal data fusion in this embodiment of the invention can execute the large model training and optimization method based on multimodal data fusion provided in this embodiment of the invention. The implementation principle is similar. The actions performed by each module and unit in the large model training and optimization device based on multimodal data fusion in each embodiment of the invention correspond to the steps in the large model training and optimization method based on multimodal data fusion in each embodiment of the invention. For detailed functional descriptions of each module of the large model training and optimization device based on multimodal data fusion, please refer to the descriptions in the corresponding large model training and optimization methods based on multimodal data fusion shown above, which will not be repeated here.
[0067] The aforementioned large model training and optimization device based on multimodal data fusion can be a computer program (including program code) running on a computer device. For example, the large model training and optimization device based on multimodal data fusion is an application software. The device can be used to execute the corresponding steps in the method provided in the embodiments of the present invention.
[0068] In some embodiments, the large model training and optimization device based on multimodal data fusion provided in this invention can be implemented using a combination of hardware and software. As an example, the large model training and optimization device based on multimodal data fusion provided in this invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the large model training and optimization method based on multimodal data fusion provided in this invention. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.
[0069] In other embodiments, the large model training and optimization device based on multimodal data fusion provided in this invention can be implemented in software. Figure 2 A large model training and optimization apparatus based on multimodal data fusion, stored in a memory, is shown. It can be software in the form of programs and plug-ins, and includes a series of modules, including an acquisition module 210, a feature alignment module 220, a fusion module 230, and a training module 240, for implementing the large model training and optimization method based on multimodal data fusion provided in the embodiments of the present invention.
[0070] The modules described in the embodiments of the present invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.
[0071] Based on the same principles as the methods shown in the embodiments of the present invention, the embodiments of the present invention also provide an electronic device, which may include, but is not limited to: a processor and a memory; the memory for storing computer programs; and the processor for executing the methods shown in any embodiment of the present invention by invoking the computer programs.
[0072] In one alternative embodiment, an electronic device is provided, such as Figure 3 As shown, Figure 3 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.
[0073] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.
[0074] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0075] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.
[0076] The memory 4003 stores application code (computer program) for executing the present invention, and its execution is controlled by the processor 4001. The processor 4001 executes the application code stored in the memory 4003 to implement the content shown in the foregoing method embodiments.
[0077] Among these, electronic devices can also be terminal devices. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.
[0078] This invention provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the corresponding content in the aforementioned method embodiments.
[0079] According to another aspect of the present invention, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various embodiments described above.
[0080] Computer program code for performing the operations of this invention can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0081] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0082] The computer-readable storage medium provided in this invention can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0083] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.
[0084] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.
Claims
1. A method for training and optimizing large models based on multimodal data fusion, characterized in that, include: Acquire multimodal data and extract the initial features corresponding to the data of each modality. For the data of each modality, the initial features corresponding to the modality include global features and multiple local features. Based on the initial features corresponding to the data of all modalities, local alignment features and global alignment features corresponding to the data of each modality are extracted. The local alignment features represent the semantically related local features in the data of different modalities, and the global alignment features represent the semantically related global features in the data of different modalities. Based on the local and global alignment features corresponding to the data of each modality, determine the aligned features corresponding to each modality; Based on the aligned features corresponding to all modalities, the initial model is trained to obtain a large multimodal model. The large multimodal model is then used to process the multimodal data to be processed, thereby obtaining the semantics corresponding to the multimodal data to be processed.
2. The method according to claim 1, characterized in that, The step of extracting local alignment features corresponding to the data of each modality based on the initial features corresponding to the data of all modalities includes: Project all local features corresponding to the data of each modality onto the same dimension to obtain the first intermediate feature corresponding to each local feature; Determine the first semantic correlation between every two first intermediate features in the data of each modality; For each local feature in each modality, based on the first semantic relevance corresponding to the local feature, determine the local features corresponding to other modalities that are semantically related to the local feature, and based on the local features corresponding to other modalities that are semantically related to the local feature, determine the aligned local features corresponding to the local feature; For each modality, the local alignment features corresponding to the modality are obtained based on the aligned local features corresponding to all local features of the modality.
3. The method according to claim 1, characterized in that, The step of extracting global alignment features for each modality's data based on the initial features corresponding to all modalities' data includes: Project the global features corresponding to all modal data onto the same dimension to obtain the second intermediate feature corresponding to each modality; Determine the second semantic relevance between the second intermediate features corresponding to each pair of modalities; For each modality, based on all the second semantic relevances corresponding to the modality, global features corresponding to other modalities that are related to the global feature semantics of the modality are determined, and based on the global features corresponding to other modalities that are related to the global feature semantics of the modality, global alignment features corresponding to the modality are determined.
4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Based on the aligned features corresponding to each modality and the weights corresponding to each modality, the aligned features corresponding to all modalities are weighted and fused to obtain the fused features; The step of training the initial model based on the aligned features corresponding to all modalities to obtain a multimodal large model includes: The initial model is trained based on the fusion features to obtain the multimodal large model.
5. The method according to any one of claims 1 to 3, characterized in that, The multimodal data includes multimodal data corresponding to different scenarios. Multimodal data corresponding to the same scenario is taken as positive samples, and multimodal data corresponding to different scenarios is taken as negative samples. Then, the aligned features corresponding to all modalities include the aligned features corresponding to all positive samples and the aligned features corresponding to all negative samples. The step of training the initial model based on the aligned features corresponding to all modalities to obtain a multimodal large model includes: Based on the aligned features corresponding to all positive samples and the aligned features corresponding to all negative samples, a contrastive learning loss value is determined, which characterizes the difference between the aligned features corresponding to positive samples and the aligned features corresponding to negative samples. Based on the aligned features corresponding to all modalities, a cross-modal projection loss value is determined, which characterizes the semantic differences between the aligned features corresponding to different modalities. Based on the weights corresponding to each mode, a dynamic regularization term is determined, wherein the dynamic regularization term characterizes the difference between the weights corresponding to each mode and the ideal weights; The initial model is trained based on the contrastive learning loss value, the cross-modal projection loss value, and the dynamic regularization term to obtain a multimodal large model.
6. A large model training and optimization device based on multimodal data fusion, characterized in that, include: The acquisition module is used to acquire multimodal data and extract the initial features corresponding to the data of each modality. For the data of each modality, the initial features corresponding to the modality include global features and multiple local features. The feature alignment module is used to extract local alignment features and global alignment features corresponding to the data of each modality based on the initial features corresponding to the data of all modalities. The local alignment features represent the semantically related local features in the data of different modalities, and the global alignment features represent the semantically related global features in the data of different modalities. The fusion module is used to determine the aligned features for each modality based on the local and global alignment features corresponding to the data of each modality. The training module is used to train the initial model based on the aligned features corresponding to all modalities to obtain a large multimodal model, and to process the multimodal data to be processed based on the large multimodal model to obtain the semantics corresponding to the multimodal data to be processed.
7. The apparatus according to claim 6, characterized in that, When the feature alignment module extracts the local alignment features corresponding to the data of each modality based on the initial features corresponding to the data of all modalities, it is specifically used for: Project all local features corresponding to the data of each modality onto the same dimension to obtain the first intermediate feature corresponding to each local feature; Determine the first semantic correlation between every two first intermediate features in the data of each modality; For each local feature in each modality, based on the first semantic relevance corresponding to the local feature, determine the local features corresponding to other modalities that are semantically related to the local feature, and based on the local features corresponding to other modalities that are semantically related to the local feature, determine the aligned local features corresponding to the local feature; For each modality, the local alignment features corresponding to the modality are obtained based on the aligned local features corresponding to all local features of the modality.
8. The apparatus according to claim 6, characterized in that, When the feature alignment module extracts the global alignment features corresponding to the data of each modality based on the initial features corresponding to the data of all modalities, it is specifically used for: Project the global features corresponding to all modal data onto the same dimension to obtain the second intermediate feature corresponding to each modality; Determine the second semantic relevance between the second intermediate features corresponding to each pair of modalities; For each modality, based on all the second semantic relevances corresponding to the modality, global features corresponding to other modalities that are related to the global feature semantics of the modality are determined, and based on the global features corresponding to other modalities that are related to the global feature semantics of the modality, global alignment features corresponding to the modality are determined.
9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method of any one of claims 1-5.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1-5.