Cross-modal feature alignment method and device, equipment, storage medium and program product
By employing dynamic weight adjustment and semantic conflict detection mechanisms, the problem of semantic conflict in cross-modal feature alignment is solved, enabling more efficient and accurate multimodal interaction and a feature alignment method that adapts to complex scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-17
- Publication Date
- 2026-03-31
AI Technical Summary
Existing cross-modal feature alignment methods are ineffective in addressing semantic conflicts in multimodal data, resulting in low interaction accuracy of multimodal agents.
Visual, speech, and text features are mapped to a shared feature space using a multilayer perceptron. Feature alignment is performed based on a dynamic weight adjustment mechanism. A semantic conflict detection and correction module is introduced. A Transformer encoder is used for context modeling and reinforcement learning to optimize the feature alignment strategy.
It improves the accuracy and robustness of feature alignment, ensures the consistency and accuracy of aligned features, enhances the interaction accuracy of multimodal agents, and adapts to complex dynamic interaction scenarios.
Smart Images
Figure CN121765629A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a cross-modal feature alignment method, apparatus, device, storage medium, and program product. Background Technology
[0002] In the interaction of multimodal agents, feature extraction and feature alignment of different modal data (such as image data, voice data, and text data) are crucial.
[0003] Currently, cross-modal feature alignment methods are mainly divided into three types: joint representation methods based on feature embedding, interaction modeling methods based on dynamic attention mechanisms, and semantic alignment methods based on contrastive learning. Joint representation methods based on feature embedding have significant shortcomings in feature alignment, semantic conflict handling, dynamic scene adaptability, and multimodal asymmetric correlation modeling, making it difficult for multimodal agents to achieve efficient and accurate multimodal interaction in complex interactive scenarios. Interaction modeling methods based on dynamic attention mechanisms mainly achieve soft feature alignment by designing cross-modal attention networks, but their robustness in dealing with multimodal semantic conflicts is insufficient, and they are prone to incorrect attention distribution due to contradictions between image data and text descriptions, resulting in low interaction accuracy. Semantic alignment methods based on contrastive learning also face problems such as insufficient negative sample sampling coverage and difficulty in adapting to temporal dependencies in dynamic interaction scenarios, leading to low interaction accuracy.
[0004] In summary, existing cross-modal feature alignment methods are ineffective in addressing semantic conflicts in multimodal data, resulting in poor feature alignment performance and low interaction accuracy of multimodal agents. Summary of the Invention
[0005] This application provides a cross-modal feature alignment method, apparatus, device, storage medium, and program product to solve the technical problem that existing cross-modal feature alignment methods are unable to effectively cope with semantic conflicts in multimodal data, resulting in poor feature alignment performance and low interaction accuracy of multimodal intelligent agents.
[0006] In a first aspect, embodiments of this application provide a cross-modal feature alignment method, comprising: inputting multimodal interaction data into a feature extraction module of a multimodal agent to perform feature extraction, obtaining visual features, speech features, and text features output by the feature extraction module; inputting the visual features, speech features, and text features into a feature alignment module of the multimodal agent to perform feature alignment, obtaining a fused feature vector output by the feature alignment module; the feature alignment module performs feature alignment based on a dynamic weight adjustment mechanism; and inputting the fused feature vector into a semantic conflict detection and correction module of the multimodal agent to perform semantic conflict detection, obtaining corrected features output by the semantic conflict detection and correction module.
[0007] In one embodiment, the fused feature vector is generated by the feature alignment module based on the following steps: mapping visual features, speech features, and text features to a shared feature space based on a multilayer perceptron to obtain mapped visual features, mapped speech features, and mapped text features; determining visual modality weights, speech modality weights, and text modality weights based on the feature similarity between the mapped visual features, mapped speech features, and mapped text features; and performing normalization processing based on the mapped visual features and visual modality weights, mapped speech features and speech modality weights, and mapped text features and text modality weights to generate the fused feature vector.
[0008] In one embodiment, the corrected features are generated by the semantic conflict detection and correction module based on the following steps: performing semantic conflict detection on the fused feature vector to obtain a semantic conflict detection result; if the semantic conflict detection result indicates the existence of a semantic conflict, then extracting features from the context information of the multimodal interaction data to generate a first context feature; and based on the first context feature, performing feature correction on the fused feature vector to obtain the corrected features.
[0009] In one embodiment, the multimodal agent further includes an interaction optimization module; the interaction optimization module optimizes the feature alignment strategy based on the following steps: generating second context features by performing context modeling through a Transformer encoder based on the corrected features; generating a feature alignment strategy by using a reinforcement learning method based on the second context features; and updating the visual modality weights, speech modality weights, and text modality weights generated by the feature alignment module based on the feature alignment strategy.
[0010] In one embodiment, the multimodal agent is trained based on a preset loss function; wherein the preset loss function is determined based on contrastive learning loss and correction loss, and the contrastive learning loss is determined based on the similarity of positive sample pairs and the similarity of negative sample pairs in the training dataset.
[0011] In one embodiment, the feature extraction module includes a visual feature extraction module, a speech feature extraction module, and a text feature extraction module. The process involves inputting multimodal interaction data into the feature extraction module of the multimodal agent, performing feature extraction, and obtaining visual features, speech features, and text features output by the feature extraction module. This includes: inputting multimodal interaction data into the visual feature extraction module, performing visual feature extraction, and obtaining visual features output by the visual feature extraction module; inputting multimodal interaction data into the speech feature extraction module, performing speech feature extraction, and obtaining speech features output by the speech feature extraction module; and inputting multimodal interaction data into the text feature extraction module, performing text feature extraction, and obtaining text features output by the text feature extraction module.
[0012] Secondly, embodiments of this application provide a cross-modal feature alignment device, comprising: a feature extraction submodule, used to input multimodal interaction data into the feature extraction module of the multimodal agent, perform feature extraction, and obtain visual features, speech features, and text features output by the feature extraction module; a feature alignment submodule, used to input the visual features, speech features, and text features into the feature alignment module of the multimodal agent, perform feature alignment, and obtain a fused feature vector output by the feature alignment module; the feature alignment module performs feature alignment based on a dynamic weight adjustment mechanism; and a feature correction submodule, used to input the fused feature vector into the semantic conflict detection and correction module of the multimodal agent, perform semantic conflict detection, and obtain corrected features output by the semantic conflict detection and correction module.
[0013] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the cross-modal feature alignment methods described above.
[0014] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements any of the cross-modal feature alignment methods described above.
[0015] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements any of the cross-modal feature alignment methods described above.
[0016] The cross-modal feature alignment method, apparatus, device, storage medium, and program product provided in this application first extract features from multimodal interaction data, extracting visual features, speech features, and text features. Then, based on a dynamic weight adjustment mechanism, the visual features, speech features, and text features are aligned to generate a fused feature vector. Unlike existing methods that forcibly align different modal features, feature alignment through a dynamic weight adjustment mechanism avoids information loss caused by forced feature alignment, which helps improve the accuracy and robustness of feature alignment and optimizes the feature alignment effect. Semantic conflict detection and correction are performed on the fused feature vector to obtain corrected features. This can promptly correct semantic conflicts between different modal features in the fused feature vector, preventing the propagation of semantically conflicting features in the multimodal agent. This solves the problem of the lack of an effective semantic conflict detection and correction mechanism in the prior art, which helps ensure the consistency and accuracy of the aligned features, thereby improving the interaction accuracy of the multimodal agent. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 This is one of the flowcharts illustrating the cross-modal feature alignment method provided in this application embodiment.
[0019] Figure 2 This is the second flowchart of the cross-modal feature alignment method provided in the embodiments of this application.
[0020] Figure 3 This is a schematic diagram of the cross-modal feature alignment device provided in the embodiments of this application.
[0021] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0022] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0023] Please see Figure 1 and Figure 2 , Figure 1 This is one of the flowcharts illustrating the cross-modal feature alignment method provided in this application. Figure 2 This is the second flowchart illustrating the cross-modal feature alignment method provided in this application. Figure 1 As shown in the embodiments of this application, the cross-modal feature alignment method includes steps S110 to S130, and the specific steps are as follows: S110: Input the multimodal interaction data into the feature extraction module of the multimodal agent to perform feature extraction and obtain the visual features, speech features and text features output by the feature extraction module.
[0024] like Figure 2 As shown, the multimodal intelligent agent in this embodiment includes a feature extraction module, a feature alignment module, a semantic conflict detection and correction module, and an interaction optimization module.
[0025] The feature extraction module includes a visual feature extraction module, a speech feature extraction module, and a text feature extraction module, which are used to extract features from different modalities.
[0026] Specifically, multimodal interaction data includes image data such as images or video frames, speech data, and text data. This multimodal interaction data is input into the visual feature extraction module of the multimodal agent. The visual feature extraction module performs preprocessing on the image data, such as normalization and scaling, and uses pre-trained convolutional neural networks (CNNs) to extract visual features. , , Represents a set of features.
[0027] In convolutional neural networks, feature vectors are typically extracted from the penultimate layer of the network, and the feature dimension is usually 2048.
[0028] Optionally, after performing preprocessing such as normalization and scaling on image data such as images or video frames, the visual feature extraction module also needs to perform data augmentation processing.
[0029] Similarly, the multimodal interaction data is input into the speech feature extraction module of the multimodal agent. The speech feature extraction module can perform preprocessing on the speech waveform of the speech data, such as segmentation, noise reduction, and spectrogram transformation, and then use a recurrent neural network (RNN) or Transformer encoder to extract speech features and obtain speech features. , .
[0030] Optionally, the RNN uses LSTM (Long Short-Term Memory) or GRU (Gated Recurrent Unit), and its feature vector dimension is usually 512 or 1024; the feature vector dimension of the output layer of the Transformer encoder is usually 768 or 1024.
[0031] Similarly, when multimodal interaction data is input into the text feature extraction module, the module can perform preprocessing such as word segmentation on the text data, and use a pre-trained language model to extract text features to obtain the text features. , .
[0032] Optionally, the language model is a BERT (Bidirectional Encoder Representations from Transformers) model, whose feature vector dimension is typically 768 (BERT-Base model) or 1024 (BERT-Large model).
[0033] These features will serve as the basis for subsequent feature alignment and optimization.
[0034] S120: Input visual features, speech features, and text features into the feature alignment module of the multimodal agent to perform feature alignment and obtain the fused feature vector output by the feature alignment module.
[0035] The feature alignment module performs feature alignment based on a dynamic weight adjustment mechanism.
[0036] Specifically, visual features, speech features, and text features are input into the feature alignment module of the multimodal agent. The feature alignment module can align features of different modalities (i.e., visual features, speech features, and text features) through a dynamic weight adjustment mechanism: first, the features of different modalities are mapped to a shared feature space, and then the weights of different modal features are dynamically adjusted according to the similarity between the features of different modalities to achieve more accurate feature alignment and generate an aligned fused feature vector.
[0037] S130: Input the fused feature vector into the semantic conflict detection and correction module of the multimodal agent to perform semantic conflict detection and obtain the corrected features output by the semantic conflict detection and correction module.
[0038] Specifically, the fused feature vector is input into the semantic conflict detection and correction module of the multimodal agent to perform semantic conflict detection and determine whether there is a semantic conflict between different modal features in the fused feature vector.
[0039] If the semantic conflict detection result indicates the presence of a semantic conflict, the fused feature vector is corrected using the contextual information of the multimodal interaction data to generate corrected features, thereby ensuring the consistency and accuracy of features across different modalities.
[0040] The cross-modal feature alignment method provided in this application first extracts features from multimodal interaction data, extracting visual, speech, and text features. Then, based on a dynamic weight adjustment mechanism, it aligns the visual, speech, and text features to generate a fused feature vector. Unlike existing methods that forcibly align different modal features, this method avoids information loss caused by forced feature alignment, improves the accuracy and robustness of feature alignment, and optimizes the alignment effect. Semantic conflict detection and correction are performed on the fused feature vector to obtain corrected features. This timely correction of semantic conflicts between different modal features in the fused feature vector prevents the propagation of semantically conflicting features in the multimodal agent, solving the problem of the lack of effective semantic conflict detection and correction mechanisms in existing technologies. This helps ensure the consistency and accuracy of the aligned features, thereby improving the interaction accuracy of the multimodal agent.
[0041] In some embodiments, the fused feature vector is generated by the feature alignment module based on the following steps: mapping visual features, speech features, and text features to a shared feature space based on a multilayer perceptron to obtain mapped visual features, mapped speech features, and mapped text features; determining visual modality weights, speech modality weights, and text modality weights based on the feature similarity between the mapped visual features, mapped speech features, and mapped text features; and performing normalization processing based on the mapped visual features and visual modality weights, mapped speech features and speech modality weights, and mapped text features and text modality weights to generate the fused feature vector.
[0042] To generate the fused feature vector, the feature alignment module needs to perform feature embedding, which refers to mapping features from different modalities to a shared feature space.
[0043] Specifically, visual features Speech features and text features After being input into the feature alignment module of the multimodal agent, the feature alignment module can utilize a multi-layer perceptron (MLP) to align visual features. Speech features and text features Mapping to a shared feature space yields the mapped feature vectors, i.e., mapped visual features. Mapping speech features and mapping text features .
[0044] Optionally, visual features Speech features and text features Each feature dimension is mapped using an independent multilayer perceptron.
[0045] Alternatively, the mapping process can be represented by the following formula: ; ; ; in, , and These are three independent multilayer perceptrons.
[0046] Furthermore, based on mapping visual features Mapping speech features and mapping text features Based on feature similarity, dynamically determine visual modality weights. Speech modal weights and text modal weights .
[0047] Specifically, the cosine similarity between different modal mapping features is first calculated, and then a similarity matrix between different modal mapping features is constructed based on the cosine similarity between different modal mapping features, which is used for subsequent weight adjustment.
[0048] The formula for calculating the cosine similarity between features of different modal mappings is as follows: ; in, Represents mapping features , Represents mapping features , and Mapping features for two different modalities, for example It can represent the mapping of visual features , It can represent the features of the mapped text. ; Mapping features and mapping features Cosine similarity between them.
[0049] After determining the cosine similarity between mapping features of different modalities, the mapping visual features are calculated based on the attention mechanism. Corresponding visual modality weights Mapping speech features Corresponding speech modal weights and mapping text features Corresponding text modal weights .
[0050] The formulas for calculating the mode weights of different mode mapping features are as follows: ; in, It is modal The weight.
[0051] Furthermore, the softmax function is used to map visual features. and visual modality weights Mapping speech features and speech modal weights and mapping text features and text modal weights Normalization is performed to generate a fused feature vector.
[0052] Among them, the fused feature vector The calculation formula is as follows: .
[0053] As can be seen from the above formula, the normalization process is a weighted summation process.
[0054] Preferably, in order to further improve the accuracy of feature alignment, after obtaining the fused feature vector, a contrastive learning optimization strategy can be used to further optimize the fused feature vector.
[0055] Specifically, the feature alignment module's ability to discriminate and optimize fused feature vectors is enhanced through contrastive learning methods, ensuring semantic consistency of the aligned fused feature vectors.
[0056] For example, during the training phase, multiple positive sample pairs and multiple negative sample pairs are constructed; the feature alignment module is trained with the goal of maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs; the loss function of the feature alignment module is a contrastive loss function. The calculation formula is as follows: ; in, This represents the preset similarity threshold.
[0057] In some embodiments, the corrected features are generated by the semantic conflict detection and correction module based on the following steps: performing semantic conflict detection on the fused feature vector to obtain a semantic conflict detection result; if the semantic conflict detection result indicates the existence of a semantic conflict, then extracting features from the context information of the multimodal interaction data to generate a first context feature; and based on the first context feature, performing feature correction on the fused feature vector to obtain the corrected features.
[0058] The semantic conflict detection and correction module is designed to detect and correct semantic conflicts between different modalities after multimodal feature alignment, ensuring the consistency and accuracy of features.
[0059] In obtaining the fused feature vector Next, the feature vectors will be fused. The input is fed into the semantic conflict detection and correction module, which can process the fused feature vector. Perform semantic conflict detection and obtain semantic conflict detection results.
[0060] Specifically, the semantic conflict detection and correction module can obtain information from the fused feature vectors. The mapping features of different modalities are extracted, and a similarity matrix between the mapping features of different modalities is constructed based on the cosine similarity between the mapping features of different modalities, which is used for semantic conflict detection.
[0061] Preset similarity threshold If any cosine similarity in the similarity matrix is lower than the similarity threshold... If so, then a semantic conflict is determined.
[0062] Optionally, the formula for calculating semantic conflict detection is as follows: ; in, This represents the semantic conflict detection result, which is a Boolean value.
[0063] Furthermore, if the semantic conflict detection result indicates the existence of a semantic conflict, then the contextual information of the multimodal interaction data is used to extract features and generate the first contextual features.
[0064] Specifically, if the semantic conflict detection result indicates the presence of a semantic conflict, the Transformer encoder is used to perform context modeling on the multimodal interaction data, extract features from the contextual information of the multimodal interaction data, capture the temporal and semantic relationships between multimodal features, and generate the first contextual features. .
[0065] Furthermore, based on the first context features The fused feature vector is then modified to obtain the modified features. .
[0066] Among them, the modified features The calculation formula is as follows: ; in, It is a correction coefficient used to control the first context feature. and conflict characteristics The weights of (i.e., different modal mapping features with semantic conflicts).
[0067] Optionally, correction factor It can be determined experimentally, and the value is usually between 0.5 and 0.8.
[0068] Preferably, a feedback mechanism is introduced to feed back the corrected features to the feature alignment module, so that the feature alignment module can dynamically adjust the feature alignment strategy according to the corrected features.
[0069] Optionally, the semantic conflict detection and correction module can continuously optimize the correction strategy through reinforcement learning to maximize the accuracy of the corrected features in the interaction: define a reward signal R, optimize the performance of the corrected features in the interaction based on the reward signal R, and continuously update the correction strategy using the policy gradient method.
[0070] In some embodiments, the multimodal agent further includes an interaction optimization module; the interaction optimization module optimizes the feature alignment strategy based on the following steps: generating second context features by performing context modeling through a Transformer encoder based on the modified features; generating a feature alignment strategy by using a reinforcement learning method based on the second context features; and updating the visual modality weights, speech modality weights, and text modality weights generated by the feature alignment module based on the feature alignment strategy.
[0071] The multimodal agent also includes an interaction optimization module, which dynamically adjusts the feature alignment strategy based on the interaction context to optimize the interaction accuracy of the multimodal agent.
[0072] Specifically, after obtaining the corrected features, the interaction optimization module can perform contextual modeling on the corrected features through the Transformer encoder to capture the temporal and semantic relationships in multimodal interactions and generate second contextual features.
[0073] The Transformer encoder uses a multi-head self-attention mechanism to capture global dependencies between multimodal features, adds positional encoding to multimodal features to preserve temporal information, and performs feature transformation through a feedforward neural network to construct a context sequence containing historical interaction information. And then based on the context sequence Generate second context features .
[0074] Furthermore, based on the second context features Feature alignment strategies are generated through reinforcement learning methods. To maximize interaction accuracy.
[0075] Specifically, the second context features Inputting the input into the policy network defined by the reinforcement learning method yields the feature alignment policy output by the policy network. Feature alignment strategy This includes weight adjustment strategies and feature fusion strategies.
[0076] Optionally, reinforcement learning methods allow defining a reward signal R to reward the performance of the interaction results of the feature alignment strategy, thereby optimizing the accuracy and consistency of the interaction results.
[0077] Optionally, rewards are divided into immediate rewards and cumulative rewards; immediate rewards refer to the immediate reward calculation for each interaction result through the reward signal R; cumulative rewards refer to the long-term strategy effect being evaluated through the cumulative evaluation of the reward signal R.
[0078] Optionally, reinforcement learning methods use policy gradient methods to update the network parameters of the policy network.
[0079] Optionally, the policy gradient method is an ε-greedy policy.
[0080] Furthermore, based on feature alignment strategy The feature alignment module updates the visual modal weights, speech modal weights, and text modal weights generated by the feature alignment module. This dynamic weight adjustment and feedback mechanism ensures that the feature alignment strategy can adapt to complex dynamic interaction scenarios.
[0081] Specifically, the features output by the policy network are aligned with the policy. Feedback is sent to the feature alignment module, based on the feature alignment strategy. The weight adjustment strategy and feature fusion strategy in the module update the visual modal weights, speech modal weights and text modal weights generated by the feature alignment module, forming a feedback loop and strategy iteration.
[0082] Optionally, the interaction accuracy of the optimized multimodal agent and the robustness of the evaluation strategy of the multimodal agent in complex dynamic scenarios are evaluated through a test set, and the optimized feature alignment results and interaction accuracy improvement curves are output for comparison and reference.
[0083] In some embodiments, the multimodal agent is trained based on a preset loss function; wherein the preset loss function is determined based on contrastive learning loss and correction loss, and the contrastive learning loss is determined based on the similarity of positive sample pairs and the similarity of negative sample pairs in the training dataset.
[0084] In this embodiment, a preset loss function is used. The calculation formula is as follows: ; in, The contrastive learning loss (i.e., the contrastive loss function of the feature alignment module) is: The correction loss is for the semantic conflict detection and correction module; These are the balancing parameters.
[0085] In some embodiments, the feature extraction module includes a visual feature extraction module, a speech feature extraction module, and a text feature extraction module; inputting multimodal interaction data into the feature extraction module of the multimodal agent for feature extraction to obtain visual features, speech features, and text features output by the feature extraction module includes: inputting multimodal interaction data into the visual feature extraction module for visual feature extraction to obtain visual features output by the visual feature extraction module; inputting multimodal interaction data into the speech feature extraction module for speech feature extraction to obtain speech features output by the speech feature extraction module; and inputting multimodal interaction data into the text feature extraction module for text feature extraction to obtain text features output by the text feature extraction module.
[0086] Specifically, multimodal interaction data includes image data such as images or video frames, speech data, and text data. This multimodal interaction data is input into the visual feature extraction module of the multimodal agent. The visual feature extraction module performs preprocessing on the image data, such as normalization and scaling, and uses pre-trained convolutional neural networks (CNNs) to extract visual features. , , Represents a set of features.
[0087] In convolutional neural networks, feature vectors are typically extracted from the penultimate layer of the network, and the feature dimension is usually 2048.
[0088] Optionally, after performing preprocessing such as normalization and scaling on image data such as images or video frames, the visual feature extraction module also needs to perform data augmentation processing.
[0089] Similarly, the multimodal interaction data is input into the speech feature extraction module of the multimodal agent. The speech feature extraction module can perform preprocessing on the speech waveform of the speech data, such as segmentation, noise reduction, and spectrogram transformation, and then use a recurrent neural network (RNN) or Transformer encoder to extract speech features and obtain speech features. , .
[0090] Optionally, the RNN uses LSTM (Long Short-Term Memory) or GRU (Gated Recurrent Unit), and its feature vector dimension is usually 512 or 1024; the feature vector dimension of the output layer of the Transformer encoder is usually 768 or 1024.
[0091] Similarly, when multimodal interaction data is input into the text feature extraction module, the module can perform preprocessing such as word segmentation on the text data, and use a pre-trained language model to extract text features to obtain the text features. , .
[0092] Optionally, the language model is a BERT (Bidirectional Encoder Representations from Transformers) model, whose feature vector dimension is typically 768 (BERT-Base model) or 1024 (BERT-Large model).
[0093] The cross-modal feature alignment method provided in this application has at least the following advantages compared with the prior art: (1) High dynamic adaptability: Through context modeling and reinforcement learning optimization strategies, the feature alignment strategy can be adjusted in real time to adapt to complex dynamic interaction scenarios, ensuring the continuity and accuracy of interaction. This solves the problem of insufficient adaptability of existing technologies in dynamic interaction scenarios, especially in multi-turn dialogues or complex interaction scenarios.
[0094] (2) Good semantic conflict handling effect: The introduction of semantic conflict detection and correction mechanism can monitor and correct semantic contradictions between multimodal features in real time, prevent the propagation of erroneous features, solve the problem that existing technologies lack effective semantic conflict detection and correction mechanisms, and effectively ensure the consistency and accuracy of multimodal features.
[0095] (3) Improved feature alignment accuracy: By using dynamic weight adjustment mechanism and asymmetric correlation modeling, information loss caused by forced alignment of feature space is avoided; the dynamic weight adjustment mechanism dynamically allocates weights according to the similarity between different modal features, and the asymmetric correlation modeling dynamically adjusts the relationship between the main and auxiliary modalities, which can effectively improve the accuracy and robustness of feature alignment.
[0096] (4) Computational efficiency optimization: Through context-adaptive optimization and multi-module collaborative work, the computational overhead of feature alignment and conflict correction is reduced. The interactive optimization module can dynamically adjust the feature alignment strategy, ensuring that the system can still operate efficiently when processing large-scale data.
[0097] This application also provides a cross-modal feature alignment device. Please refer to... Figure 3 , Figure 3 This is a schematic diagram of the cross-modal feature alignment device provided in an embodiment of this application. In this embodiment, the cross-modal feature alignment device includes a feature extraction submodule 310, a feature alignment submodule 320, and a feature correction submodule 330.
[0098] The feature extraction submodule 310 is used to input multimodal interaction data into the feature extraction module of the multimodal agent, perform feature extraction, and obtain visual features, speech features, and text features output by the feature extraction module.
[0099] The feature alignment submodule 320 is used to input visual features, speech features and text features into the feature alignment module of the multimodal agent, perform feature alignment, and obtain the fused feature vector output by the feature alignment module.
[0100] The feature alignment module performs feature alignment based on a dynamic weight adjustment mechanism.
[0101] The feature correction submodule 330 is used to input the fused feature vector into the semantic conflict detection and correction module of the multimodal agent to perform semantic conflict detection and obtain the corrected features output by the semantic conflict detection and correction module.
[0102] In some embodiments, the fused feature vector is generated by the feature alignment module based on the following steps: mapping visual features, speech features, and text features to a shared feature space based on a multilayer perceptron to obtain mapped visual features, mapped speech features, and mapped text features; determining visual modality weights, speech modality weights, and text modality weights based on the feature similarity between the mapped visual features, mapped speech features, and mapped text features; and performing normalization processing based on the mapped visual features and visual modality weights, mapped speech features and speech modality weights, and mapped text features and text modality weights to generate the fused feature vector.
[0103] In some embodiments, the corrected features are generated by the semantic conflict detection and correction module based on the following steps: performing semantic conflict detection on the fused feature vector to obtain a semantic conflict detection result; if the semantic conflict detection result indicates the existence of a semantic conflict, then extracting features from the context information of the multimodal interaction data to generate a first context feature; and based on the first context feature, performing feature correction on the fused feature vector to obtain the corrected features.
[0104] In some embodiments, the multimodal agent further includes an interaction optimization module; the interaction optimization module optimizes the feature alignment strategy based on the following steps: generating second context features by performing context modeling through a Transformer encoder based on the modified features; generating a feature alignment strategy by using a reinforcement learning method based on the second context features; and updating the visual modality weights, speech modality weights, and text modality weights generated by the feature alignment module based on the feature alignment strategy.
[0105] In some embodiments, the multimodal agent is trained based on a preset loss function; wherein the preset loss function is determined based on contrastive learning loss and correction loss, and the contrastive learning loss is determined based on the similarity of positive sample pairs and the similarity of negative sample pairs in the training dataset.
[0106] In some embodiments, the feature extraction module includes a visual feature extraction module, a speech feature extraction module, and a text feature extraction module; inputting multimodal interaction data into the feature extraction module of the multimodal agent for feature extraction to obtain visual features, speech features, and text features output by the feature extraction module includes: inputting multimodal interaction data into the visual feature extraction module for visual feature extraction to obtain visual features output by the visual feature extraction module; inputting multimodal interaction data into the speech feature extraction module for speech feature extraction to obtain speech features output by the speech feature extraction module; and inputting multimodal interaction data into the text feature extraction module for text feature extraction to obtain text features output by the text feature extraction module.
[0107] This application also provides an electronic device. Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 4 As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a cross-modal feature alignment method.
[0108] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0109] This application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the cross-modal feature alignment methods provided by the above methods.
[0110] This application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the cross-modal feature alignment methods provided by the above methods.
[0111] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0112] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A cross-modal feature alignment method, characterized in that, include: Multimodal interaction data is input into the feature extraction module of the multimodal agent for feature extraction, and visual features, speech features and text features output by the feature extraction module are obtained. The visual features, speech features, and text features are input into the feature alignment module of the multimodal agent to perform feature alignment and obtain the fused feature vector output by the feature alignment module. The feature alignment module performs feature alignment based on a dynamic weight adjustment mechanism; The fused feature vector is input into the semantic conflict detection and correction module of the multimodal agent to perform semantic conflict detection and obtain the corrected features output by the semantic conflict detection and correction module.
2. The cross-modal feature alignment method according to claim 1, characterized in that, The fused feature vector is generated by the feature alignment module based on the following steps: Based on a multilayer perceptron, the visual features, the speech features, and the text features are mapped to a shared feature space to obtain mapped visual features, mapped speech features, and mapped text features. Based on the feature similarity between the mapped visual features, the mapped speech features, and the mapped text features, visual modality weights, speech modality weights, and text modality weights are determined. Based on the mapped visual features and the visual modality weights, the mapped speech features and the speech modality weights, and the mapped text features and the text modality weights, normalization processing is performed to generate the fused feature vector.
3. The cross-modal feature alignment method according to claim 1, characterized in that, The corrected feature is generated by the semantic conflict detection and correction module based on the following steps: Semantic conflict detection is performed on the fused feature vector to obtain semantic conflict detection results; If the semantic conflict detection result indicates the existence of a semantic conflict, then the context information of the multimodal interaction data is used to extract features and generate a first context feature. Based on the first contextual features, the fused feature vector is modified to obtain the modified features.
4. The cross-modal feature alignment method according to claim 2, characterized in that, The multimodal intelligent agent also includes an interaction optimization module; The interaction optimization module optimizes the feature alignment strategy based on the following steps: Based on the corrected features, context modeling is performed using a Transformer encoder to generate second context features; Based on the second contextual features, a feature alignment strategy is generated using a reinforcement learning method. Based on the feature alignment strategy, the visual modality weights, speech modality weights, and text modality weights generated by the feature alignment module are updated.
5. The cross-modal feature alignment method according to claim 1, characterized in that, The multimodal agent is trained based on a preset loss function; The preset loss function is determined based on the contrastive learning loss and the correction loss. The contrastive learning loss is determined based on the similarity of positive sample pairs and the similarity of negative sample pairs in the training dataset.
6. The cross-modal feature alignment method according to claim 1, characterized in that, The feature extraction module includes a visual feature extraction module, a speech feature extraction module, and a text feature extraction module; The step of inputting multimodal interaction data into the feature extraction module of the multimodal agent for feature extraction, and obtaining the visual features, speech features, and text features output by the feature extraction module, includes: The multimodal interaction data is input into the visual feature extraction module to extract visual features and obtain the visual features output by the visual feature extraction module. The multimodal interaction data is input into the speech feature extraction module to extract speech features and obtain the speech features output by the speech feature extraction module. The multimodal interaction data is input into the text feature extraction module to extract text features, thereby obtaining the text features output by the text feature extraction module.
7. A cross-modal feature alignment device, characterized in that, include: The feature extraction submodule is used to input multimodal interaction data into the feature extraction module of the multimodal agent, perform feature extraction, and obtain the visual features, speech features, and text features output by the feature extraction module. The feature alignment submodule is used to input the visual features, the speech features, and the text features into the feature alignment module of the multimodal agent to perform feature alignment and obtain the fused feature vector output by the feature alignment module. The feature alignment module performs feature alignment based on a dynamic weight adjustment mechanism; The feature correction submodule is used to input the fused feature vector into the semantic conflict detection and correction module of the multimodal agent to perform semantic conflict detection and obtain the corrected features output by the semantic conflict detection and correction module.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the cross-modal feature alignment method as described in any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the cross-modal feature alignment method as described in any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the cross-modal feature alignment method as described in any one of claims 1 to 6.