Multi-modal semantic recognition method and system based on cross-modal contrast learning

Abstract

The application discloses a multi-modal semantic recognition method and system based on cross-modal contrast learning, belongs to the technical field of artificial intelligence and multi-modal semantic recognition, and solves the problems of mode quality difference and difficult sample recognition in multi-modal semantic recognition.The method comprises the following steps: S1, data preprocessing: standardizing and preprocessing visual, speech and text multi-modal data; S2, multi-modal feature extraction: extracting feature representations of visual, speech and text modes respectively; S3, cross-modal contrast learning: mapping the multi-modal features to a unified space, and enhancing the semantic consistency between modes through contrast learning; S4, adaptive mode fusion: dynamically adjusting the fusion weight according to the mode quality, and filtering noise features through a gating mechanism; S5, joint optimization: constructing a joint loss function of classification loss and contrast loss; S6, semantic recognition: outputting a target category recognition result; and S7, difficult sample mining: identifying difficult samples based on prediction uncertainty and strengthening training.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence and multimodal semantic recognition technology, specifically involving a multimodal semantic recognition method and system based on cross-modal contrastive learning, which is particularly suitable for scenarios that integrate multiple modal information such as vision, speech, and text for high-precision semantic classification and recognition. Background Technology

[0002] With the rapid development of human-computer interaction technology, semantic recognition and understanding have become a research hotspot in the field of artificial intelligence. Multimodal semantic recognition, by integrating multiple information sources such as visual information, speech features, and text semantics, can more comprehensively and accurately understand the high-level semantic state and interaction intentions of humans. However, existing multimodal semantic recognition methods still face many technical challenges.

[0003] First, existing methods fail to fully utilize the complementarity and consistency constraints between different modalities. Traditional multimodal fusion methods mostly employ simple feature concatenation or weighted averaging, neglecting the semantic connections and mutual promotion between modalities. This makes it difficult for the model to learn deep semantic representations across modalities, especially when the quality of a particular modality is poor or missing, resulting in a significant drop in overall semantic recognition performance. How to effectively establish consistency constraints between modalities, enabling the representations of different modalities to align with each other in a shared semantic space, remains an urgent problem to be solved.

[0004] Secondly, the modality fusion process lacks an adaptive evaluation mechanism for the quality and reliability of each modality. In practical applications, the data quality varies significantly across different modalities, such as visual information degradation under low light conditions, speech information distortion in noisy environments, and semantic ambiguity caused by colloquial text. Existing fusion methods often assign fixed weights or simple attention scores to each modality, failing to dynamically adjust the fusion strategy based on the real-time quality of the modalities. This mechanical fusion approach easily introduces noise from low-quality modalities into the final semantic representation, reducing the system's robustness to complex scenarios. Designing an adaptive modality fusion mechanism that dynamically adjusts weights based on the reliability of each modality and suppresses noise information is crucial for improving the practicality of multimodal semantic understanding systems.

[0005] Furthermore, existing methods have significant shortcomings in handling complex and difficult semantic samples. Multimodal semantic recognition tasks commonly suffer from class imbalance and ambiguous semantic boundaries. Some high-level semantic categories are highly similar in expression and have a limited number of samples, making it easy for models to confuse these semantic boundary regions. Traditional training strategies typically treat all samples equally, leading to overfitting on easily distinguishable samples while underlearning for semantically ambiguous or boundary samples. The lack of effective mining and enhancement mechanisms for complex semantic samples severely restricts the model's ability to model subtle semantic differences. How to identify and enhance the learning of difficult semantic samples and improve the model's ability to discriminate semantic boundaries is a major bottleneck in improving overall performance.

[0006] Finally, existing technologies lack a unified optimization framework to jointly model semantic classification tasks and cross-modal consistency constraints. Most methods treat multimodal feature extraction, modality fusion, and semantic recognition as independent modules trained in series, failing to achieve end-to-end joint optimization. This separate training paradigm limits information interaction and collaborative learning between modules, making it difficult to fully leverage the overall advantages of multimodal data. The core issue in achieving high-performance multimodal semantic recognition is how to construct a unified optimization objective that enhances cross-modal semantic consistency while ensuring semantic recognition accuracy, and improves overall system performance through joint optimization.

[0007] Therefore, there is an urgent need in this field for a new multimodal semantic recognition technology based on cross-modal contrastive learning to solve the above problems. Summary of the Invention

[0008] To overcome the shortcomings of existing technologies, the present invention aims to provide a multimodal semantic recognition method and system based on cross-modal contrastive learning. By constructing a unified cross-modal representation space and a dynamic fusion mechanism, it solves the problems of modality quality differences and difficult sample recognition in multimodal semantic recognition, and provides an end-to-end optimization scheme for highly robust semantic recognition in practical application scenarios. When using only publicly available multimodal datasets, it can effectively establish consistency constraints between modalities through cross-modal contrastive learning, and support adaptive fusion and difficult sample recognition.

[0009] On the one hand, this invention provides a multimodal semantic recognition method based on cross-modal contrastive learning, comprising the following steps:

[0010] (a) Step S1, Data Preprocessing: Obtain the multimodal dataset and preprocess the multimodal data; the multimodal data includes visual modal data, speech modal data and text modal data; Step S2, Multimodal Feature Extraction: Construct a multimodal feature extraction network to extract features from the preprocessed visual modal data, speech modal data, and text modal data respectively, and obtain visual feature vectors, speech feature vectors, and text feature vectors; Step S3, Cross-modal contrastive learning: Construct a cross-modal contrastive learning framework, and map the visual feature vector, speech feature vector, and text feature vector to a unified contrastive learning feature space through the corresponding projection head network to obtain projected features; Step S4, Adaptive Modality Fusion: Design an adaptive modality fusion module to dynamically adjust the fusion weights based on each modality feature, and calculate the quality score of each modality feature through a quality assessment network; Step S5, Loss Function Construction: Construct the total loss function and jointly optimize semantic recognition and cross-modal consistency; Step S6, Classification and Recognition: The multimodal data to be identified is processed according to steps S1 to S4. After obtaining the fused features, the probability distribution of the target category is output through the classifier, and the category with the highest probability is selected as the final recognition result. Step S7, Hard Sample Mining: Introduce a hard sample mining strategy, input the gated fused features into the classifier, obtain the predicted probability distribution of each category, and calculate the entropy value of the predicted probability distribution.

[0011] Furthermore, in step S1, the multimodal dataset is a publicly available multimodal dataset, including MELD, CMU-MOSEI, and CMU-MOSI datasets; the visual modal data is a video frame sequence or image sequence containing the target; the speech modal data is the corresponding audio signal; and the text modal data is the corresponding text transcription content or dialogue text.

[0012] Furthermore, in step S2, the visual feature extraction uses ResNet-101 as the backbone network, and the calculation process of the channel attention mechanism is as follows: First, the convolutional feature map Global average pooling is used to obtain channel descriptors. ,in, ; Then, channel attention weights are calculated using a two-layer fully connected network and an activation function: ,in, and For learnable parameters, To reduce the proportion, It is the ReLU activation function. Use the Sigmoid activation function; Finally, the attention weights are multiplied by the original feature map. ,in, This represents channel-level broadcast multiplication, resulting in a recalibrated feature map. Global average pooling is performed on F' to obtain the visual feature vector. .

[0013] Furthermore, in step S2, the specific process of speech feature extraction is as follows: performing a short-time Fourier transform (STFT) on the preprocessed speech signal to obtain the time spectrum. The Mel spectrum is calculated using a Mel filter bank on the time spectrum; then, logarithmic operations are performed on the Mel spectrum followed by Discrete Cosine Transform (DCT) to extract the Mel frequency cepstral coefficients; finally, prosodic features, including the fundamental frequency, are extracted. Short-term energy Zero crossing rate The mellitudinal cepstral coefficient features and prosodic features are concatenated to obtain a temporal feature sequence. The hidden state sequence is then encoded using a bidirectional long short-term memory network (Bi-LSTM) and average pooled to obtain a speech feature vector.

[0014] Furthermore, in step S2, text feature extraction uses a BERT pre-trained language model to convert the text sequence into a token sequence, adding a key at the beginning of the sequence. A special marker is added to the end of the sequence to obtain the input sequence. The token sequence is converted into a vector representation through an embedding layer. The input vector sequence is then passed through an L-layer Transformer encoder to extract the last layer. Mark the corresponding hidden state As a semantic representation of the text sequence, or by pooling the hidden states of all tokens, a fixed-dimensional text feature vector is obtained through dimensional transformation using a fully connected layer.

[0015] Further, in step S3, the projection head network is a multilayer perceptron (MLP). For each sample in the training batch, positive sample pairs and negative sample pairs are constructed. Different modal features of the same sample are used as positive sample pairs, and modal features of different samples are used as negative sample pairs. The cross-modal contrast loss function is calculated. For each modal combination, the contrast loss is calculated using a cosine similarity function and a temperature coefficient-controlled Softmax function. The total cross-modal contrast loss is the average of the contrast losses of the three modal combinations.

[0016] Furthermore, in step S4, the quality evaluation network includes a fully connected layer and a sigmoid activation function. The quality score is normalized by the Softmax function to obtain the fusion weights of each modality. The fusion features are obtained by weighted summation, and then a gating mechanism is introduced to adaptively filter the fusion features.

[0017] Further, in step S5, the total loss function includes weighted cross-entropy loss and cross-modal contrastive learning loss. The classification loss and contrastive learning loss are adjusted by a balancing coefficient. The weighted cross-entropy loss is defined as: , in, For the sample One-hot encoding of the real label, To predict probabilities, Difficult sample weights; the total loss function is defined as: ,in, This is the balance coefficient.

[0018] Furthermore, in step S7, hard sample mining also includes modality inconsistency detection, using a single-modality classifier to analyze visual features. Speech features Text features Classify the modalities to obtain the predicted probability distributions for each modality. , , ; Calculate the KL divergence between modal predictions: , Calculate the average KL divergence of the three pairs of modes: , when When, it is determined to be a modally inconsistent sample, among which The inconsistency threshold is set; for modally inconsistent samples, the loss weight is set as follows: ,in, This is the scaling factor.

[0019] On the other hand, the present invention provides a multimodal semantic recognition system based on cross-modal contrastive learning, for performing the above-described multimodal semantic recognition method based on cross-modal contrastive learning, including: Data preprocessing module: used to acquire multimodal datasets and perform preprocessing, including object detection and alignment, speech denoising and endpoint detection, and text segmentation and cleaning; Multimodal feature extraction module: used to extract features from visual modal data, speech modal data, and text modal data, including visual feature extraction submodule, speech feature extraction submodule, and text feature extraction submodule; Cross-modal contrastive learning module: includes a projection head submodule and a contrastive loss calculation submodule; the projection head submodule maps the features of each modality to a unified contrastive learning space; the contrastive loss calculation submodule constructs positive and negative sample pairs and calculates the contrastive loss; Adaptive fusion module: used to dynamically adjust the fusion weights according to each modality feature, and calculate the quality score of each modality feature through the quality assessment network, including a quality assessment submodule, a weight calculation submodule and a gating submodule; Loss calculation module: used to calculate the total loss function, including weighted cross-entropy loss, cross-modal contrastive loss, and regularization loss; Semantic recognition output module: includes a fully connected layer and a Softmax classifier, used to output the probability distribution of each category, and select the category with the highest probability as the recognition result; Hard Sample Mining Module: Used to calculate the entropy value of the predicted probability distribution and assign higher loss weights to hard samples, including an entropy calculation submodule and a weight adjustment submodule.

[0020] Compared with the prior art, the present invention has the following advantages: (1) This invention, through a cross-modal contrastive learning framework, aligns feature representations of different modalities in a unified semantic space, effectively enhancing the consistency constraints between modalities. By constructing positive and negative sample pairs and calculating the contrastive loss, the modal features of different samples are brought closer to each other in the representation space, while the modal features of different samples are moved further apart, thereby learning more discriminative cross-modal shared representations. This end-to-end joint optimization mechanism significantly improves the model's ability to comprehensively utilize multimodal information, and even when the quality of a certain modality is poor, it can still maintain a high recognition accuracy through complementary information from other modalities.

[0021] (2) This invention designs an adaptive modality fusion mechanism, which dynamically calculates the reliability score of each modality feature through a quality assessment network and adaptively adjusts the fusion weights accordingly. Compared with traditional fixed weights or simple attention mechanisms, this method can dynamically adjust based on real-time modality quality, automatically reducing the contribution of low-quality modalities and strengthening the weights of high-quality modalities. The gating mechanism further refines the filtering of the fused features, effectively suppressing noise components and ensuring the robustness of the final representation. This mechanism enables the system to have stronger adaptability and stability when facing complex and ever-changing real-world application scenarios.

[0022] (3) This invention introduces an entropy-based hard sample mining strategy, which quantifies the difficulty of samples by calculating the entropy value of the predicted probability distribution and automatically identifies hard samples with high model prediction uncertainty. Higher loss weights are assigned to the identified hard samples, guiding the model to pay more attention to learning these boundary samples during training. This strategy effectively alleviates class imbalance and ambiguous boundary problems, significantly improving the model's ability to distinguish complex categories. Simultaneously, combined with a modal inconsistency detection mechanism, it further enhances the ability to handle multimodal information conflict samples.

[0023] (4) This invention constructs a unified end-to-end optimization framework that organically integrates multimodal feature extraction, cross-modal contrastive learning, adaptive fusion, and semantic recognition, and achieves collaborative optimization of each module through a joint loss function. Weighted cross-entropy loss ensures classification accuracy, cross-modal contrastive loss enhances semantic consistency, and the balance coefficient flexibly adjusts the relative importance of the two. This joint optimization paradigm avoids the information loss and optimization fragmentation problems caused by traditional serial training, enabling each module to promote each other and improve collaboratively, ultimately achieving the optimization of overall performance. Attached Figure Description

[0024] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a system block diagram of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0026] Example 1 like Figure 1 As shown, this embodiment provides a multimodal semantic recognition method based on cross-modal contrastive learning, which specifically includes the following steps: (i) Step S1, Data Preprocessing: Obtain a multimodal dataset, which includes visual modal data, speech modal data and text modal data; and preprocess the multimodal data, perform object detection and alignment on the visual modal data, extract the target region and normalize it to a uniform size; perform resampling, noise reduction and endpoint detection on the speech modal data; and perform word segmentation, stop word removal and punctuation mark removal on the text modal data.

[0027] The multimodal datasets are publicly available multimodal datasets, including MELD, CMU-MOSEI, and CMU-MOSI datasets. Visual modal data consists of video frame sequences or image sequences containing targets, represented as... ,in, For the first Frame image, and These represent the height and width of the image, respectively; the speech modal data is the corresponding audio signal, represented as... ,in, For the first Each audio sampling point; text modal data consists of the corresponding transcribed text or dialogue text, represented as... ,in, For the first A word or token.

[0028] (ii) Step S2, Multimodal Feature Extraction: Construct a multimodal feature extraction network to extract features from the preprocessed visual, speech, and text modal data respectively; visual feature extraction uses a convolutional neural network as the backbone network, and introduces a channel attention mechanism after the convolutional layer to adaptively adjust the weights of different feature channels and output visual feature vectors; speech feature extraction uses a bidirectional long short-term memory network to process the Mel frequency cepstral coefficient sequence and prosodic features, captures the dynamic changes of speech through temporal modeling, and outputs speech feature vectors; text feature extraction uses a pre-trained language model to encode the text sequence, extract contextual semantic information, and extract fixed-dimensional text feature vectors through pooling layers or specific labels.

[0029] (1) Visual feature extraction The convolutional neural network for visual feature extraction uses ResNet-101 as the backbone network. The computation process of the channel attention mechanism is as follows: First, the convolutional feature map Perform global average pooling to obtain channel descriptors. ,in, ; Then, channel attention weights are calculated using a two-layer fully connected network and an activation function. ,in, and For learnable parameters, To reduce the proportion, It is the ReLU activation function. Use the Sigmoid activation function; Finally, the attention weights are multiplied by the original feature map. ,in, This represents channel-level broadcast multiplication, resulting in a recalibrated feature map. Global average pooling is performed on F' to obtain the visual feature vector. .

[0030] (2) Speech feature extraction The specific process of speech feature extraction includes: ① Perform a Short Time Fourier Transform (STFT) on the preprocessed speech signal to obtain the time spectrum. Apply a Mel filter bank to the time spectrum to calculate the Mel spectrum: ,in, For the first Frequency response of a Mel filter.

[0031] ② Perform logarithmic operations on the Mel spectrum and then perform Discrete Cosine Transform (DCT) to extract the Mel frequency cepstral coefficients: ,in, The number of Mel filters, This is the index for cepstral coefficients.

[0032] ③ Extract prosodic features, including fundamental frequency. Short-term energy Zero crossing rate The temporal feature sequence is obtained by concatenating MFCC features and prosodic features. ,in, For time steps, For feature dimensions.

[0033] ④ Using a bidirectional long short-term memory (Bi-LSTM) network for encoding, the forward hidden state is: Backward hidden state: ;Splitting bidirectional hidden state: .

[0034] ⑤ Perform average pooling on the hidden state sequence To obtain the speech feature vector ,in, is the dimension of the LSTM hidden layer.

[0035] (3) Text feature extraction Text feature extraction uses the BERT pre-trained language model. The text sequence is converted into a token sequence, and a token is added to the beginning of the sequence. A special marker is added to the end of the sequence to obtain the input sequence. The token sequence is converted into a vector representation through an embedding layer, with the input embedding being... ,in, For word embedding, For location embedding, This is segment embedding. The input vector sequence passes through an L-layer Transformer encoder, and the first... The output of the layer is Among them, Embed the input; extract the last layer. Mark the corresponding hidden state As a semantic representation of the text sequence, or as a pooling of the hidden state of all tokens. Dimensional transformation is performed through fully connected layers. This yields fixed-dimensional text feature vectors. ,in, and These are learnable parameters.

[0036] (III) Step S3, cross-modal contrastive learning: Construct a cross-modal contrastive learning framework, map the visual feature vector, speech feature vector, and text feature vector to a unified contrastive learning feature space through the corresponding projection head network to obtain projected features; for each sample in the training batch, construct positive sample pairs and negative sample pairs, take the different modal features of the same sample as positive sample pairs, and take the modal features of different samples as negative sample pairs; calculate the cross-modal contrastive loss function. For each pair of modal combinations, calculate the contrastive loss through the cosine similarity function and the Softmax function controlled by the temperature coefficient. The total cross-modal contrastive loss is the average of the contrastive losses of the three pairs of modal combinations.

[0037] The projection head network is a multilayer perceptron (MLP), consisting of two fully connected layers. The calculation formula is as follows: , in, For modality eigenvectors ( , or ), and The projector features are learnable parameters, with ReLU as the activation function; the projected features are L2 normalized. For samples in the training batch Construct a set of positive sample pairs Cross-modal contrastive loss function for mode pairs Defined as: , in, For batch size, The cosine similarity function is used. The temperature coefficient ranges from 0.05 to 0.2; the total transmodal contrast loss is: .

[0038] Cross-modal contrastive learning also includes a hard-to-bear sample sampling strategy, for samples modality Calculate its projection features Compared with all other samples in the batch modality Projection features similarity Select the top similarity rankings Individual and category labels The samples are used as the set of hard-to-bear samples. ,in, Assign weight factors to hard-to-bear samples. The corrected contrast loss is: , in, For positive sample pairs, For a simple negative sample set, For a difficult negative sample set.

[0039] (iv) Step S4, Adaptive Modality Fusion: Design an adaptive modality fusion module to dynamically adjust the fusion weights based on the quality and reliability of each modality feature; calculate the quality score of each modality feature through a quality assessment network, which includes a fully connected layer and a sigmoid activation function; normalize the quality score using the Softmax function to obtain the fusion weights of each modality; obtain the fusion features through weighted summation; introduce a gating mechanism to adaptively filter the fusion features. The gating mechanism generates a gating weight vector through learnable parameters and performs element-wise multiplication with the fusion features to automatically suppress low-quality or noisy feature components.

[0040] The quality assessment network calculates a quality score for each modal feature. ,in, It is the Sigmoid activation function. and For learnable parameters, Representing modes The quality score is calculated; the quality score is normalized using the Softmax function to obtain the fusion weights. ,in, The fusion features are obtained by weighted summation. .

[0041] The gate function is defined as ,in, and For learnable parameters, Here is the gate weight vector; the gated features are: ,in, This represents element-wise multiplication. This results in the final fusion features. Adaptive modality fusion also includes a cross-modal attention mechanism for modality pairs. modality Features as queries Modality Features as keys Sum Calculate cross-attention: , in, , , For learnable projection matrices, The dimension is the key vector; for the combination of three modes , , Calculate cross-attention separately to obtain enhanced features. , , Perform a residual connection between the cross-attention output and the original features. ,in, This is a layer normalization operation; and the enhanced features are... , , Conduct quality assessment and weighted fusion.

[0042] (v) Step S5, loss function construction: construct the total loss function and jointly optimize the classification and cross-modal consistency; the total loss function includes weighted cross-entropy loss and cross-modal contrastive learning loss, and adjusts the relative importance of classification loss and contrastive learning loss by balancing coefficient.

[0043] The weighted cross-entropy loss is defined as: , in, For the sample One-hot encoding of the real label, To predict probabilities, Difficult sample weights; the total loss function is defined as: , in, This is the balance coefficient.

[0044] (vi) Step S6, Classification and Recognition: For the multimodal data to be identified, process it according to steps S1 to S5, obtain the fused features, output the probability distribution of the categories through the classifier, and select the category with the highest probability as the final recognition result.

[0045] (vii) Step S7, Difficult Sample Mining: Introduce a difficult sample mining strategy to improve the model's ability to identify difficult samples; input the gated fusion features into the classifier to obtain the predicted probability distribution of each category; calculate the entropy value of the predicted probability distribution to measure the uncertainty of the prediction. The higher the entropy value, the more uncertain the prediction and the more difficult the sample; set an entropy threshold to determine difficult samples. The entropy threshold is dynamically set according to the number of categories; assign higher loss weights to difficult samples to enhance the model's learning of difficult samples.

[0046] Fusion features The input is fed into a classifier, and the predicted probability distribution is obtained through a fully connected layer and a softmax function:

[0047] in, , , To belong to category The probability satisfies ; Calculate the entropy value of the predicted probability distribution. ,in, , Total number of categories. Set entropy threshold. ,in, This is the threshold coefficient; for the sample Its cross-entropy loss weight is defined as: , in, Difficult sample weighting factors.

[0048] Hard sample mining also includes modality inconsistency detection, using unimodal classifiers to analyze visual features. Speech features Text features Classify the modalities to obtain the predicted probability distributions for each modality. , , ; Calculate the KL divergence between modal predictions: .

[0049] Calculate the average KL divergence of the three pairs of modes: , when When the condition is met, it is determined to be a modally inconsistent sample. The inconsistency threshold is set; for modally inconsistent samples, the loss weight is set to... ,in, This is the scaling factor.

[0050] This invention introduces an entropy-based hard sample mining strategy. By calculating the entropy value of the predicted probability distribution, the difficulty of a sample is quantified, automatically identifying hard samples with high model prediction uncertainty. These identified hard samples are assigned higher loss weights, guiding the model to focus more on learning these boundary samples during training. This strategy effectively alleviates class imbalance and ambiguous boundary problems, significantly improving the model's ability to distinguish complex data. Furthermore, combined with a modality inconsistency detection mechanism, it further enhances the ability to handle samples with conflicting multimodal information.

[0051] Example 2 This embodiment is a preferred embodiment, providing a multimodal semantic recognition method based on cross-modal contrastive learning, which includes the following steps: (a) Step S1, Data Preprocessing The MELD multimodal dataset was chosen as the training and testing data. This dataset contains 1433 dialogues with a total of 13708 sentences, covering seven basic emotion categories (neutral, happy, sad, angry, surprised, fearful, and disgusted). Each sample includes a video clip, the corresponding audio, and a text transcription.

[0052] For the visual modality, the MTCNN face detection algorithm is used to detect face regions in video frames. Affine transformations are performed to align the face to the eyes and nose, and a 224×224 pixel facial region is cropped. The image is then normalized, with pixel values ​​scaled to the [0,1] range and standardized. For each video segment, 16 frames are uniformly sampled as the visual input sequence.

[0053] For the speech modality, the original audio is resampled to a 16kHz sampling rate, spectral subtraction is used for noise reduction, and a dual-threshold endpoint detection algorithm is used to extract valid speech segments. The first 5 seconds of each speech segment are retained, and any insufficient portions are filled with silence.

[0054] For text modalities, the WordPiece tokenizer is used for word segmentation, removing stop words and punctuation marks while retaining content words and semantically relevant vocabulary. The text sequence is truncated or padded to a maximum length of 128 tokens, and special markers [CLS] and [SEP] are added.

[0055] (ii) Step S2, Multimodal Feature Extraction Visual feature extraction uses a ResNet-101 pre-trained on ImageNet as the backbone network. A 224×224×3 facial image is input, and after passing through convolutional layers, batch normalization layers, and ReLU activation layers, a 7×7×2048 feature map is obtained. A channel attention mechanism is introduced after the fourth residual block, with a reduction ratio r set to 16. Global average pooling is performed on the feature map to obtain a 2048-dimensional channel descriptor. Channel attention weights are calculated through two fully connected layers (2048→128→2048) and Sigmoid activation. These weights are then multiplied channel-by-channel with the feature map to achieve feature recalibration. Finally, global average pooling is performed on the recalibrated feature map to obtain a 2048-dimensional visual feature vector.

[0056] Speech feature extraction begins with a short-time Fourier transform (SFT) of the preprocessed audio, with a window length of 25ms and a frame shift of 10ms, yielding the time-frequency spectrum. A 40-channel Mel filter bank is then used to calculate the Mel spectrum, followed by logarithmic operations and Discrete Cosine Transform (DCT) to extract 13-dimensional MFCC features. Simultaneously, the fundamental frequency (using the YIN algorithm), short-time energy, and zero-crossing rate are extracted as prosodic features. The MFCC and prosodic features are concatenated to obtain a 16-dimensional temporal feature sequence. Encoding is performed using a bidirectional LSTM network with a hidden layer dimension of 256. The forward and backward hidden states are concatenated to obtain a 512-dimensional vector. Average pooling is then applied to the hidden states at all time steps to obtain the 512-dimensional speech feature vector.

[0057] Text feature extraction employs a BERT-base-uncased pre-trained model, containing a 12-layer Transformer encoder with a hidden layer dimension of 768. The token sequence after word segmentation is converted into a sum of word embeddings, position embeddings, and segment embeddings, which is then input into the 12-layer encoder for context modeling. The 768-dimensional hidden state corresponding to the last layer [CLS] marker is extracted and transformed through a fully connected network (768→512) to obtain a 512-dimensional text feature vector.

[0058] (III) Step S3, cross-modal comparative learning Three projection head networks are constructed to map visual, speech, and text features, respectively. Each projection head contains two fully connected layers: the first layer has a dimension of 256, and the second layer has a dimension of 128. The projected features are then subjected to L2 normalization to ensure that the magnitude of all feature vectors is 1.

[0059] In the training batch (batch size N=32), the three modal projection features for each sample i are... Construct a set of positive sample pairs. Specifically, , , As positive sample pairs, for each positive sample pair, the corresponding modal features of the other 31 samples are used as negative samples.

[0060] When calculating cross-modal contrast loss, cosine similarity is used. Temperature coefficient as a similarity measure Set to 0.1. For the visual-speech modality pair, calculate the contrast loss: , Similarly, the contrastive loss for the visual-text and speech-text modal pairs is calculated. The total contrastive loss is the average of the three.

[0061] To enhance learning on hard-to-bear samples, for each sample's modality pair, the similarity between it and all other samples from different classes within the batch is calculated, and the sample with the highest similarity is selected. Each sample is considered a hard-to-bear sample and assigned a weight factor. The formula for calculating contrast loss was revised to make the model focus more on distinguishing negative sample pairs that are difficult to separate.

[0062] (iv) Step S4, Adaptive Modality Fusion First, a cross-modal attention mechanism is applied to the three modal features. Taking visual-speech as an example, visual features are... The query vector is obtained through linear projection. (Dimension 512), speech features The key vector is obtained through linear projection. Sum value vector Calculate attention score: , We obtain the visual attention enhancement features for speech. Similarly, we calculate the cross-attention for other modal combinations. We average the multiple attention outputs from each modality, perform residual connections and layer normalization with the original features, and obtain the enhanced features.

[0063] The quality evaluation network calculates a quality score for each enhanced feature. The network structure consists of a single fully connected layer (512→1) with sigmoid activation. The fusion weights are calculated using Softmax normalization to ensure that the sum of the three weights is 1. The weighted sum is then used to obtain the fusion feature, which has a dimension of 512.

[0064] The gated network uses a fully connected layer (512→512) with sigmoid activation to generate the gated vector. The gated vector is multiplied element-wise with the fused features to achieve feature filtering, suppress noise components, and retain important information.

[0065] (v) Step S5, Loss Function Construction and Model Training The weighted cross-entropy loss is calculated, and the total loss function is: , Among them, the balance coefficient The model was trained using the AdamW optimizer with an initial learning rate of 2e-4, employing a cosine annealing learning rate scheduling strategy for 50 epochs. The batch size was 32, and gradient clipping was used to prevent gradient explosion.

[0066] (vi) Step S6, Classification and Recognition For the multimodal data to be identified, process it according to the above steps, and finally output the probability distribution of the categories. Select the category with the highest probability as the identification result.

[0067] (vii) Step S7, Difficult Sample Mining The gated fused features are input into a classifier, which is a fully connected layer (512→7) with softmax activation, resulting in predicted probability distributions for the 7 classes. The entropy of the predicted probability distributions is then calculated. , A higher entropy value indicates greater uncertainty in the prediction. Set an entropy threshold. , where the threshold coefficient ,get Regarding entropy The samples that are difficult to identify are classified as hard samples and are assigned loss weights. For simple samples, weights .

[0068] Simultaneously, modal inconsistency detection is performed. Three unimodal classifiers are used to classify visual, speech, and text features respectively, obtaining unimodal prediction probabilities. The average KL divergence between modalities is calculated. An inconsistency threshold is set. For samples exceeding the threshold, additional loss weights are added.

[0069] Experiments were conducted on the MELD dataset, which was divided into training, validation, and test sets in a 7:1:2 ratio. Weighted accuracy and weighted F1 score were used as evaluation metrics.

[0070] Comparative experimental results show that the method of this invention achieves significant improvements compared to the baseline method: Single-modal method (visual only) accuracy 59.3%, F1 score 56.8%; Single-modal method (speech only) accuracy 61.7%, F1 score 58.2%; Single-modal method (text only) accuracy 64.1%, F1 score 61.5%; Simple feature concatenation method accuracy 66.8%, F1 score 64.2%; Attention fusion method accuracy 68.5%, F1 score 66.1%; The method of this invention (complete) accuracy 72.4%, F1 score 70.8%.

[0071] Ablation experiments validated the effectiveness of each module: accuracy decreased to 69.2% (-3.2%) after removing cross-modal contrastive learning; accuracy decreased to 68.7% (-3.7%) after removing adaptive fusion; accuracy decreased to 70.1% (-2.3%) after removing gating mechanism; accuracy decreased to 70.6% (-1.8%) after removing hard sample mining; and accuracy decreased to 71.1% (-1.3%) after removing cross-modal cross attention.

[0072] Example 3

[0073] This embodiment provides a multimodal semantic recognition system based on cross-modal contrastive learning, used to execute the methods of Embodiments 1 and 2. Figure 2 As shown, this system includes a data preprocessing module, a multimodal feature extraction module, a cross-modal contrastive learning module, an adaptive fusion module, a loss calculation and optimization module, a semantic recognition output module, and a hard sample mining module.

[0074] (1) Data preprocessing module: used to acquire multimodal datasets and perform preprocessing, including object detection and alignment, speech denoising and endpoint detection, and text segmentation and cleaning.

[0075] For visual modal data, object detection, region cropping, and alignment operations are performed to extract target regions and normalize them to a uniform size. For speech modal data, resampling, noise reduction, and endpoint detection are performed to extract valid speech segments. For text modal data, word segmentation, stop word filtering, and sequence standardization are performed to construct a token sequence of uniform length. After preprocessing, standardized multimodal data is output to provide a consistent input format for subsequent feature extraction.

[0076] (2) Multimodal feature extraction module: used to extract features from visual modal data, speech modal data and text modal data, including visual feature extraction submodule, speech feature extraction submodule and text feature extraction submodule.

[0077] Based on the standardized data output from the data preprocessing module, a multimodal feature extraction network is constructed to extract deep semantic features of each modality. The visual feature extraction submodule uses ResNet-101 as the backbone network, introduces a channel attention mechanism to adaptively adjust the feature channel weights, and outputs a fixed-dimensional visual feature vector through global average pooling. The speech feature extraction submodule performs short-time Fourier transform on the audio signal to obtain the time spectrum, applies Mel filter bank to extract MFCC features, combines prosodic features, and performs temporal encoding through a bidirectional LSTM network to output a speech feature vector. The text feature extraction submodule uses a BERT pre-trained language model to perform context encoding on the token sequence, extracts the hidden state of the [CLS] tag, and transforms it into a fixed-dimensional text feature vector through a fully connected layer.

[0078] (3) Cross-modal contrastive learning module: including projection head submodule and contrastive loss calculation submodule. The projection head submodule maps the features of each modality to a unified contrastive learning space; the contrastive loss calculation submodule constructs positive and negative sample pairs and calculates the contrastive loss to enhance the consistency constraints between modalities.

[0079] Based on the visual, speech, and text feature vectors obtained from the multimodal feature extraction module, the features of each modality are mapped to a unified contrastive learning space and L2 normalized through a projection head network. Positive and negative sample pairs are constructed for the samples in the training batch, with different modal features of the same sample as positive sample pairs and modal features of different samples as negative sample pairs. The feature similarity between modalities is calculated using cosine similarity, and the contrastive loss is calculated using a temperature-controlled Softmax function. A hard negative sample sampling strategy is introduced, which assigns higher weights to negative sample pairs with high similarity to enhance the model's ability to distinguish samples that are difficult to differentiate. The cross-modal contrastive loss is output for subsequent joint optimization.

[0080] (4) Adaptive fusion module: used to dynamically adjust the fusion weights according to each modal feature and calculate the quality score of each modal feature through a quality assessment network; including a quality assessment submodule, a weight calculation submodule and a gating submodule. The quality assessment submodule calculates the quality score of each modal feature; the weight calculation submodule dynamically adjusts the fusion weights according to the quality score; the gating submodule performs adaptive filtering on the fused features.

[0081] Based on the original features output by the multimodal feature extraction module, the interaction information between modalities is first captured through a cross-modal attention mechanism. Attention scores are calculated for each modal pair and feature enhancement is performed. Subsequently, a quality assessment submodule calculates the quality score for each modal feature and evaluates the reliability of the features using a fully connected layer and a sigmoid activation function. Dynamic fusion weights for each modality are obtained through Softmax normalization, and weighted summation is performed on the enhanced features to obtain the fused features. A gating mechanism is introduced to adaptively filter the fused features. A gating weight vector is generated through learnable parameters and multiplied element-wise with the fused features to suppress low-quality and noise components. The final fused features after gating are output for classification.

[0082] (5) Loss calculation and optimization module: used to calculate the total loss function, including weighted cross-entropy loss, cross-modal contrast loss and regularization loss.

[0083] Based on the sample weights output by the hard sample mining module and the contrastive loss output by the cross-modal contrastive learning module, a unified total loss function is constructed for end-to-end optimization. A weighted cross-entropy loss is calculated to ensure classification accuracy, assigning higher weights to the classification errors of hard samples and modality-inconsistent samples. Cross-modal contrastive loss is combined to enhance semantic consistency between modalities. The relative importance of classification loss and contrastive loss is adjusted by a balancing coefficient to achieve joint optimization of the classification task and cross-modal constraints. Gradient descent is used to update network parameters, enabling collaborative training of all modules.

[0084] (6) Semantic recognition output module: including fully connected layer and Softmax classifier, used to output the probability distribution of each category and select the category with the highest probability as the recognition result.

[0085] The model trained based on the loss calculation and optimization module performs end-to-end inference on the multimodal data to be identified. It sequentially goes through data preprocessing, feature extraction, cross-modal contrast mapping, adaptive fusion, and gating filtering to obtain the final fused feature representation. The fused features are input into the classifier, which outputs the probability distribution of each category through a fully connected layer and a Softmax activation function. The category with the highest probability is selected as the final classification and recognition result. The confidence threshold can be set according to the application scenario requirements to mark or reject low-confidence predictions to ensure the reliability of the output results.

[0086] (7) Hard Sample Mining Module: Used to calculate the entropy value of the predicted probability distribution and assign higher loss weights to hard samples; includes an entropy calculation submodule and a weight adjustment submodule. The entropy calculation submodule calculates the entropy value of the predicted probability distribution and identifies high-entropy hard samples; the weight adjustment submodule assigns higher loss weights to hard samples.

[0087] Based on the fusion features output by the adaptive fusion module, a classifier is used to obtain the predicted probability distributions for each category. The entropy value of the predicted probability distribution is calculated as a measure of sample difficulty; a higher entropy value indicates greater prediction uncertainty and higher sample recognition difficulty. An entropy threshold is dynamically set according to the number of categories, and samples with entropy values ​​exceeding the threshold are identified as difficult samples. A modality inconsistency detection mechanism is introduced, using a single-modality classifier to classify each modality feature separately, calculating the KL divergence of the predicted probability distributions between modalities, and identifying samples with conflicting multimodal information. Higher loss weights are assigned to difficult samples and modally inconsistent samples to enhance the model's learning ability for difficult and boundary samples.

[0088] In summary, this invention constructs a unified end-to-end optimization framework that organically integrates multimodal feature extraction, cross-modal contrastive learning, adaptive fusion, and semantic recognition. A joint loss function enables collaborative optimization of each module. Weighted cross-entropy loss ensures classification accuracy, cross-modal contrastive loss enhances semantic consistency, and a balance coefficient flexibly adjusts the relative importance of both. This joint optimization paradigm avoids the information loss and fragmented optimization problems caused by traditional sequential training, allowing each module to promote and synergistically improve, ultimately achieving optimal overall performance.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal semantic recognition method based on cross-modal contrastive learning, characterized in that, Includes the following steps: Step S1, Data Preprocessing: Obtain the multimodal dataset and preprocess the multimodal data; the multimodal data includes visual modal data, speech modal data, and text modal data; Step S2, Multimodal Feature Extraction: Construct a multimodal feature extraction network to extract features from the preprocessed visual modal data, speech modal data, and text modal data respectively, and obtain visual feature vectors, speech feature vectors, and text feature vectors; Step S3, Cross-modal contrastive learning: Construct a cross-modal contrastive learning framework, and map the visual feature vector, speech feature vector, and text feature vector to a unified contrastive learning feature space through the corresponding projection head network to obtain projected features; Step S4, Adaptive Modality Fusion: Design an adaptive modality fusion module to dynamically adjust the fusion weights based on each modality feature, and calculate the quality score of each modality feature through a quality assessment network; Step S5, Loss Function Construction: Construct the total loss function and jointly optimize semantic recognition and cross-modal consistency; Step S6, Classification and Recognition: The multimodal data to be identified is processed according to steps S1 to S4. After obtaining the fused features, the probability distribution of the target category is output through the classifier, and the category with the highest probability is selected as the final recognition result. Step S7, Hard Sample Mining: Introduce a hard sample mining strategy, input the gated fused features into the classifier, obtain the predicted probability distribution of each category, and calculate the entropy value of the predicted probability distribution.

2. The multimodal semantic recognition method based on cross-modal contrastive learning according to claim 1, characterized in that, In step S1, the multimodal dataset is a publicly available multimodal dataset, including MELD, CMU-MOSEI and CMU-MOSI datasets; the visual modal data is a video frame sequence or image sequence containing the target; the speech modal data is the corresponding audio signal; and the text modal data is the corresponding text transcription content or dialogue text.

3. The multimodal semantic recognition method based on cross-modal contrastive learning according to claim 1, characterized in that, In step S2, visual feature extraction uses ResNet-101 as the backbone network, and the calculation process of the channel attention mechanism is as follows: First, the convolutional feature map Global average pooling is used to obtain channel descriptors. ,in, ; Then, channel attention weights are calculated using a two-layer fully connected network and an activation function: ,in, and For learnable parameters, To reduce the proportion, It is the ReLU activation function. Use the Sigmoid activation function; Finally, the attention weights are multiplied by the original feature map. ,in, This represents channel-level broadcast multiplication, resulting in a recalibrated feature map. Global average pooling is performed on F' to obtain the visual feature vector. .

4. The multimodal semantic recognition method based on cross-modal contrastive learning according to claim 1, characterized in that, In step S2, the specific process of speech feature extraction is as follows: Short-Time Fourier Transform (STFT) is performed on the preprocessed speech signal to obtain the time spectrum. The Mel frequency spectrum is calculated by applying a Mel filter bank to the time spectrum; then the Mel frequency spectrum is logarithmically calculated and a discrete cosine transform (DCT) is performed to extract the Mel frequency cepstral coefficients. Further extraction of prosodic features includes fundamental frequency. Short-term energy Zero crossing rate The mellitudinal cepstral coefficient features and prosodic features are concatenated to obtain a temporal feature sequence. The hidden state sequence is then encoded using a bidirectional long short-term memory network (Bi-LSTM) and average pooled to obtain a speech feature vector.

5. The multimodal semantic recognition method based on cross-modal contrastive learning according to claim 1, characterized in that, In step S2, text feature extraction uses the BERT pre-trained language model to convert the text sequence into a token sequence, adding a token to the beginning of the sequence. A special marker is added to the end of the sequence to obtain the input sequence. The token sequence is converted into a vector representation through an embedding layer. The input vector sequence is then passed through an L-layer Transformer encoder to extract the last layer. Mark the corresponding hidden state As a semantic representation of the text sequence, or by pooling the hidden states of all tokens, a fixed-dimensional text feature vector is obtained through dimensional transformation using a fully connected layer.

6. The multimodal semantic recognition method based on cross-modal contrastive learning according to claim 1, characterized in that, In step S3, the projection head network is a multilayer perceptron (MLP). For each sample in the training batch, positive sample pairs and negative sample pairs are constructed. Different modal features of the same sample are used as positive sample pairs, and modal features of different samples are used as negative sample pairs. The cross-modal contrast loss function is calculated. For each pair of modal combinations, the contrast loss is calculated using the cosine similarity function and the temperature coefficient-controlled Softmax function. The total cross-modal contrast loss is the average of the contrast losses of the three pairs of modal combinations.

7. The multimodal semantic recognition method based on cross-modal contrastive learning according to claim 1, characterized in that, In step S4, the quality evaluation network includes a fully connected layer and a sigmoid activation function. The quality score is normalized by the Softmax function to obtain the fusion weights of each modality. The fusion features are obtained by weighted summation, and then a gating mechanism is introduced to adaptively filter the fusion features.

8. The multimodal semantic recognition method based on cross-modal contrastive learning according to claim 1, characterized in that, In step S5, the total loss function includes weighted cross-entropy loss and cross-modal contrastive learning loss. The classification loss and contrastive learning loss are adjusted by a balancing coefficient. The weighted cross-entropy loss is defined as: ， in, For the sample One-hot encoding of the real label, To predict probabilities, Difficult sample weights; the total loss function is defined as: ， in, This is the balance coefficient.

9. The multimodal semantic recognition method based on cross-modal contrastive learning according to claim 1, characterized in that, In step S7, hard sample mining also includes modality inconsistency detection, using a single-modality classifier to analyze visual features. Speech features Text features Classify the modalities to obtain the predicted probability distributions for each modality. , , ; Calculate the KL divergence between modal predictions: ， Calculate the average KL divergence of the three pairs of modes: ， when When, it is determined to be a modally inconsistent sample, among which, The inconsistency threshold is set; for modally inconsistent samples, the loss weight is set as follows: ,in, This is the scaling factor.

10. A multimodal semantic recognition system based on cross-modal contrastive learning, used to execute the multimodal semantic recognition method based on cross-modal contrastive learning as described in any one of claims 1-9, characterized in that, include: Data preprocessing module: used to acquire multimodal datasets and perform preprocessing, including object detection and alignment, speech denoising and endpoint detection, and text segmentation and cleaning; Multimodal feature extraction module: used to extract features from visual modal data, speech modal data, and text modal data, including visual feature extraction submodule, speech feature extraction submodule, and text feature extraction submodule; Cross-modal contrastive learning module: includes a projection head submodule and a contrastive loss calculation submodule; the projection head submodule maps the features of each modality to a unified contrastive learning space; the contrastive loss calculation submodule constructs positive and negative sample pairs and calculates the contrastive loss; Adaptive fusion module: used to dynamically adjust the fusion weights according to each modality feature, and calculate the quality score of each modality feature through the quality assessment network, including a quality assessment submodule, a weight calculation submodule and a gating submodule; Loss calculation module: used to calculate the total loss function, including weighted cross-entropy loss, cross-modal contrastive loss, and regularization loss; Semantic recognition output module: includes a fully connected layer and a Softmax classifier, used to output the probability distribution of each category, and select the category with the highest probability as the recognition result; Hard Sample Mining Module: Used to calculate the entropy value of the predicted probability distribution and assign higher loss weights to hard samples, including an entropy calculation submodule and a weight adjustment submodule.

Images