Speech recognition method based on bimodal mixed contrast enhancement, electronic device, chip, storage medium and program product

Abstract

The application provides a speech recognition method based on bimodal mixed contrast enhancement, an electronic device, a chip, a storage medium and a program product; the method comprises the following steps: acquiring multi-modal speech training data; a bimodal mixed contrast learning model is constructed and trained, wherein samples with the same emotional label under the same mode are taken as positive sample pairs, samples with different emotional labels are taken as negative sample pairs, intra-modal contrast loss is calculated to enhance the ability of the bimodal mixed contrast learning model to distinguish intra-modal fine-grained emotional features; samples with the same emotional label under different modes are taken as positive sample pairs, samples with different emotional labels under different modes are taken as negative sample pairs, inter-modal contrast loss is calculated to realize the alignment and complementarity of different modal feature spaces; the intra-modal contrast loss and the inter-modal contrast loss are fused to obtain multi-modal mixed contrast loss, and the model parameters of the bimodal mixed contrast learning model are optimized; based on the features extracted by the bimodal mixed contrast learning model, a downstream speech recognition task is trained.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and speech signal processing technology, and in particular to a speech recognition method, electronic device, chip, storage medium and program product based on dual-modal hybrid contrast enhancement, which is applicable to speech emotion recognition scenarios and speech recognition scenarios for patients with articulation disorders. Background Technology

[0002] With the development of artificial intelligence technology, speech recognition has become a key technology for human-computer interaction. Traditional speech recognition models typically rely on manually extracted acoustic features, such as Mel frequency cepstral coefficients or Mel spectrograms. These features are processed by a one-dimensional convolutional neural network and then input into a sequence-to-sequence model for recognition. However, this feature engineering method has significant shortcomings: First, the manual feature extraction process may lose some information in the original speech signal that is effective for recognition, increasing the model's dependence on manual features and limiting its end-to-end learning ability. Second, in speech emotion recognition tasks, most existing models output emotions as discrete labels, ignoring the representation of emotions in a continuous dimensional space (such as the PAD space composed of pleasure, activation, and dominance), resulting in insufficient ability to distinguish similar emotions such as happiness and pleasure, as well as weak emotions. Furthermore, the training data for existing models generally lacks pathological speech data (such as the speech of patients with dysarthria), leading to poor recognition performance of abnormal speech.

[0003] In recent years, contrastive learning techniques have demonstrated significant advantages in feature learning. Supervised contrastive learning learns more discriminative feature representations by bringing similar samples (positive sample pairs) closer together and distancing dissimilar samples (negative sample pairs) further apart in the feature space. Multimodal contrastive learning aims to fuse information from different modalities (such as speech, text, and images). For example, existing techniques include multimodal contrastive learning methods that fuse facial and speech data for identity verification. However, existing methods still suffer from modal gaps in modal alignment and tend to focus on intra-sample or intra-class interaction learning, failing to fully explore global association information across modalities, modalities, samples, and categories.

[0004] Furthermore, while speech-based models like SenseVoice can handle multilingual and multi-task speech understanding, they still have limitations in fine-grained emotion recognition and adaptability to pathological speech. Although self-supervised pre-trained models like Wav2vec 2.0 can learn effective representations from raw audio, further technological innovation is needed to organically combine these representations with the requirements of specific tasks (such as fine-grained emotion recognition and pathological speech recognition).

[0005] Therefore, there is an urgent need for a speech recognition method that can fully utilize the original speech signal information, effectively integrate multimodal features, and accurately model emotional dimensions and pathological features to solve the above-mentioned technical problems. Summary of the Invention

[0006] This application provides a speech recognition method, electronic device, chip, storage medium, and program product based on dual-modal hybrid contrast enhancement, which reduces reliance on manual features and achieves speech recognition closer to end-to-end; enhances the model's ability to recognize fine-grained emotions such as discrete emotions and dimensional emotions; improves the model's recognition accuracy for pathological speech such as articulation disorders; and fully explores and integrates information within and between multiple modalities through an innovative contrastive learning mechanism.

[0007] The technical solution of this application embodiment is implemented as follows: This application provides a speech recognition method based on dual-modal hybrid contrast enhancement, the method comprising: Acquire multimodal speech training data; the multimodal speech training data includes at least the original speech signal data and the Mel speech spectrogram data generated from the original speech signal data; A bimodal hybrid contrastive learning model is constructed and trained. Samples with the same sentiment label within the same modality are treated as positive sample pairs, and samples with different sentiment labels are treated as negative sample pairs. Intramodal contrastive loss is calculated to enhance the model's ability to distinguish fine-grained sentiment features within a single modality. Furthermore, samples with the same sentiment label in different modalities are treated as positive sample pairs, and samples with different sentiment labels in different modalities are treated as negative sample pairs. Intermodal contrastive loss is calculated to achieve alignment and complementarity of feature spaces across different modalities. The intramodal and intermodal contrastive losses are then fused to obtain a multimodal hybrid contrastive loss, and the model parameters of the bimodal hybrid contrastive learning model are optimized. Based on the features extracted by the dual-modal hybrid contrastive learning model, downstream speech recognition tasks are trained, which include at least speech emotion recognition tasks and automatic speech recognition tasks.

[0008] In some embodiments, the multimodal speech training data further includes pathological speech signal data, which is speech data of patients with dysarthria. In the intramodal and intermodal contrastive learning of the dual-modal hybrid contrastive learning model, the pathological speech signal data is treated as an independent modality and participates in the contrastive learning training together with the original speech signal data and the Mel spectrogram data, so that the normal speech feature space of the dual-modal hybrid contrastive learning model can learn and approximate the pathological speech features.

[0009] In some implementations, the speech emotion recognition task includes discrete emotion recognition tasks and dimensional emotion recognition tasks; The samples in the multimodal speech training data are all labeled with dimensional sentiment labels, which are coordinate values ​​in the PAD (P represents Pleasure-displeasure, indicating the positive or negative of the subject's emotional state; A represents Arousal-nonarousal, indicating the subject's neurophysiological activation level; D represents Dominance-submissiveness) in the three-dimensional sentiment space. In the contrastive learning training of the bimodal hybrid contrastive learning model, the contrastive loss function is dynamically adjusted using the aforementioned dimensional sentiment labels.

[0010] In some implementations, dynamically adjusting the contrast loss function using the dimensional sentiment labels includes: For any two training samples in the training batch of the bimodal hybrid contrastive learning model, calculate their spatial distance in the PAD three-dimensional emotion space; When calculating intramodal contrast loss or intermodal contrast loss, the fixed temperature coefficient in the contrast loss function is replaced with a dynamic temperature coefficient associated with the training sample pair. Specifically, for positive sample pairs, a smaller spatial distance value means that the training samples are highly similar in the sentiment dimension, and the bimodal mixture contrastive learning model pays more attention to them to bring their feature representations closer together. For negative sample pairs, a larger spatial distance value means that the samples have large sentiment differences, and the bimodal mixture contrastive learning model pays less attention to them. On the other hand, a smaller spatial distance value means that they are difficult negative samples that are hard to distinguish, which triggers the bimodal mixture contrastive learning model to apply a larger gradient penalty through a dynamic temperature coefficient, so as to force the bimodal mixture contrastive learning model to learn more refined feature discrimination.

[0011] In some implementations, the encoder of the dual-modal hybrid contrastive learning model employs a pre-trained self-supervised learning speech representation model wav2vec, and connects a projection head composed of two multilayer perceptrons after the output layer of wav2vec to map the encoded features to the low-dimensional representation space required for contrastive learning; during training, a partial adjustment strategy is adopted to update the parameters of the projection head and adjust some high-level network parameters of the encoder.

[0012] In some implementations, the downstream speech recognition task is trained using a non-autoregressive SenseVoice-small model with an encoder-only structure; The fusion features extracted by the trained bimodal hybrid contrastive learning model are concatenated with identity embedding vectors representing different tasks to form an enhanced input feature sequence. The enhanced input feature sequence is then input into the encoder of the SenseVoice-small model for multi-task joint training. The task identity embedding vectors include at least automatic speech recognition embedding and speech emotion recognition embedding.

[0013] In some implementations, during the training of downstream speech recognition tasks, the loss function for speech emotion recognition tasks is composed of the cross-entropy loss of discrete emotion classification and the mean squared error loss of dimensional emotion prediction, so as to achieve joint modeling of discrete categories and continuous dimensional attributes of emotion.

[0014] The speech recognition device based on dual-modal hybrid contrast enhancement provided in this application includes: An acquisition unit is used to acquire multimodal speech training data; the multimodal speech training data includes at least the original speech signal data and the Mel speech spectrogram data generated from the original speech signal data; A construction unit is used to build and train a bimodal hybrid contrastive learning model. This model uses samples with the same sentiment label within the same modality as positive sample pairs and samples with different sentiment labels as negative sample pairs, calculating intramodal contrastive loss to enhance the model's ability to distinguish fine-grained sentiment features within a single modality. It also uses samples with the same sentiment label in different modalities as positive sample pairs and samples with different sentiment labels in different modalities as negative sample pairs, calculating intermodal contrastive loss to achieve alignment and complementarity of feature spaces across different modalities. The intramodal and intermodal contrastive losses are then fused to obtain a multimodal hybrid contrastive loss, which is used to optimize the model parameters of the bimodal hybrid contrastive learning model. The training unit is used to train downstream speech recognition tasks based on features extracted by the bimodal hybrid contrastive learning model; the training of downstream speech recognition tasks includes at least speech emotion recognition tasks and automatic speech recognition tasks.

[0015] In some embodiments, the multimodal speech training data further includes pathological speech signal data, which is speech data of patients with dysarthria. In the intramodal and intermodal contrastive learning of the dual-modal hybrid contrastive learning model, the pathological speech signal data is treated as an independent modality and participates in the contrastive learning training together with the original speech signal data and the Mel spectrogram data, so that the normal speech feature space of the dual-modal hybrid contrastive learning model can learn and approximate the pathological speech features.

[0016] In some implementations, the speech emotion recognition task includes discrete emotion recognition tasks and dimensional emotion recognition tasks; The samples in the multimodal speech training data are all labeled with dimensional sentiment labels, which are the coordinate values ​​of the PAD in the three-dimensional sentiment space; In the contrastive learning training of the bimodal hybrid contrastive learning model, the contrastive loss function is dynamically adjusted using the aforementioned dimensional sentiment labels.

[0017] In some implementations, the construction unit dynamically adjusts the contrastive loss function using the dimensional sentiment labels, including: For any two training samples in the training batch of the bimodal hybrid contrastive learning model, calculate their spatial distance in the PAD three-dimensional emotion space; When calculating intramodal contrast loss or intermodal contrast loss, the fixed temperature coefficient in the contrast loss function is replaced with a dynamic temperature coefficient associated with the training sample pair. Specifically, for positive sample pairs, a smaller spatial distance value means that the training samples are highly similar in the sentiment dimension, and the bimodal mixture contrastive learning model pays more attention to them to bring their feature representations closer together. For negative sample pairs, a larger spatial distance value means that the samples have large sentiment differences, and the bimodal mixture contrastive learning model pays less attention to them. On the other hand, a smaller spatial distance value means that they are difficult negative samples that are hard to distinguish, which triggers the bimodal mixture contrastive learning model to apply a larger gradient penalty through a dynamic temperature coefficient, so as to force the bimodal mixture contrastive learning model to learn more refined feature discrimination.

[0018] In some implementations, the encoder of the dual-modal hybrid contrastive learning model employs a pre-trained self-supervised learning speech representation model wav2vec, and connects a projection head composed of two multilayer perceptrons after the output layer of wav2vec to map the encoded features to the low-dimensional representation space required for contrastive learning; during training, a partial adjustment strategy is adopted to update the parameters of the projection head and adjust some high-level network parameters of the encoder.

[0019] In some implementations, the downstream speech recognition task is trained using a non-autoregressive SenseVoice-small model with an encoder-only structure; The fusion features extracted by the trained bimodal hybrid contrastive learning model are concatenated with identity embedding vectors representing different tasks to form an enhanced input feature sequence. The enhanced input feature sequence is then input into the encoder of the SenseVoice-small model for multi-task joint training. The task identity embedding vectors include at least automatic speech recognition embedding and speech emotion recognition embedding.

[0020] In some implementations, during the training of downstream speech recognition tasks, the loss function of the training unit for speech emotion recognition tasks is composed of the cross-entropy loss of discrete emotion classification and the mean squared error loss of dimensional emotion prediction, so as to achieve joint modeling of discrete categories and continuous dimensional attributes of emotion.

[0021] The electronic device provided in this application includes a processor and a memory. The memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the above-described speech recognition method based on bimodal hybrid contrast enhancement.

[0022] The chip provided in this application includes a processor for calling and running a computer program from a memory, causing a device equipped with the chip to perform the above-described speech recognition method based on dual-modal hybrid contrast enhancement.

[0023] The computer-readable storage medium provided in this application embodiment is used to store a computer program that causes a computer to execute the above-described speech recognition method based on bimodal hybrid contrast enhancement.

[0024] The computer program product provided in this application includes computer program instructions that cause a computer to execute the above-described speech recognition method based on bimodal hybrid contrast enhancement.

[0025] The embodiments of this application have the following beneficial effects: Using Log-Mel spectrogram features as a data augmentation method, and comparing them with the original speech signal to form bimodal data for training, eliminates the need for direct input of the Log-Mel spectrogram, avoiding the loss of effective information in the original speech signal, reducing reliance on handcrafted features, and allowing direct input of the original speech signal during the inference stage, thus achieving end-to-end speech recognition; In the contrastive learning stage, pathological speech is used as an important data source. Through the design of a bimodal hybrid contrastive loss function, normal speech signals can learn the feature information of pathological speech, narrowing the distance between the two in the feature space, effectively improving the recognition accuracy of speech from patients with articulation disorders, and expanding the applicability of the speech recognition model; The dimensional affective index (PAD) is integrated into the contrastive loss function, replacing the traditional fixed temperature coefficient, and mean squared error (MSE) is introduced in downstream tasks. The Error (MSE) loss enables the model to learn discrete and dimensional emotion features simultaneously, enhancing its ability to distinguish between similar and weak emotions and achieving more granular emotion recognition. By constructing a bimodal hybrid contrastive loss function through intramodal and intermodal contrastive learning, it can fully explore the subtle features and key information within each modality, while effectively capturing cross-modal correlations and fusing information between modalities and samples, significantly improving the model's recognition performance and generalization ability. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating the speech recognition method based on dual-modal hybrid contrast enhancement provided in an embodiment of this application; Figure 2 This is a schematic diagram of the composition structure of the speech recognition framework based on dual-modal hybrid contrast enhancement provided in the embodiments of this application; Figure 3 This is a schematic diagram of the composition structure of the speech recognition device based on dual-modal hybrid contrast enhancement provided in the embodiments of this application; Figure 4 This is a schematic structural diagram of an electronic device provided in an embodiment of this application; Figure 5 This is a schematic structural diagram of the chip according to an embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0028] The purpose of the technical solution in this application is to improve the accuracy, generalization ability and applicability of speech recognition by using SenseVoice bimodal hybrid contrast enhancement speech recognition, based on bimodal hybrid contrast learning, dimensional emotion fusion and training with pathological speech data.

[0029] Figure 1 This is a flowchart illustrating the speech recognition method based on dual-modal hybrid contrast enhancement provided in this application embodiment, as shown below. Figure 1 As shown, the speech recognition method based on dual-modal hybrid contrast enhancement in this application specifically includes the following processing steps: Step 101: Obtain multimodal speech training data.

[0030] In this embodiment of the application, the multimodal speech training data includes at least the original speech signal data and the Mel speech spectrogram data generated from the original speech signal data.

[0031] This step is the data preparation stage. The speech training data in this embodiment covers three types of core data: raw speech signals, Log-Mel Spectrogram feature data, and pathological speech signal data of patients with articulation disorders. To meet the training requirements for emotion recognition, each speech data in the speech training dataset is labeled with discrete emotion labels (such as happy, angry, sad, etc.) and dimensional emotion PAD indicators (P represents pleasure, A represents arousal, and D represents dominance).

[0032] As an example, the logic for obtaining Mel spectrogram feature data is as follows: 80-dimensional log-mel features are calculated on the original speech signal, and after stacking consecutive frames, they are downsampled by 6 times to obtain the Mel spectrogram feature sequence. This feature sequence will be used as an important modality in bimodal contrastive learning to complement the information of the original speech signal and the pathological speech signal.

[0033] Mel spectrogram features are obtained by performing short-time Fourier transform and Mel filter bank processing on the original speech signal data. In this embodiment, it is regarded as an enhanced mode derived from the original signal.

[0034] Step 102: Construct and train a bimodal hybrid contrastive learning model.

[0035] Samples with the same sentiment label within the same modality are treated as positive sample pairs, and samples with different sentiment labels are treated as negative sample pairs. Intramodal contrastive loss is calculated to enhance the ability of the bimodal hybrid contrastive learning model to distinguish fine-grained sentiment features within a single modality. Furthermore, samples with the same sentiment label in different modalities are treated as positive sample pairs, and samples with different sentiment labels in different modalities are treated as negative sample pairs. Intermodal contrastive loss is calculated to achieve alignment and complementarity of feature spaces in different modalities. The intramodal contrastive loss and the intermodal contrastive loss are fused to obtain a multimodal hybrid contrastive loss, and the model parameters of the bimodal hybrid contrastive learning model are optimized.

[0036] In this embodiment, during intra-modal and inter-modal contrastive learning of the dual-modal hybrid contrastive learning model, the pathological speech signal data can be treated as an independent modality and participated in contrastive learning training together with the original speech signal data and the Mel spectrogram data. This allows the normal speech feature space of the dual-modal hybrid contrastive learning model to learn and approximate the pathological speech features. The speech emotion recognition task includes discrete emotion recognition and dimensional emotion recognition tasks. Samples in the multimodal speech training data are all labeled with dimensional emotion tags, which are the coordinate values ​​of pleasure, activation, and dominance (PAD) in the three-dimensional emotion space. During the contrastive learning training of the dual-modal hybrid contrastive learning model, the contrastive loss function is dynamically adjusted using the dimensional emotion tags.

[0037] For any two training samples in a training batch of the bimodal mixture contrastive learning model, calculate their spatial distance in the PAD 3D sentiment space. When calculating intramodal or intermodal contrastive loss, replace the fixed temperature coefficient in the contrastive loss function with a dynamic temperature coefficient associated with the training sample pair. For positive sample pairs, a smaller spatial distance value means that the training samples are highly similar in the sentiment dimension, and the bimodal mixture contrastive learning model gives more attention to them to bring their feature representations closer. For negative sample pairs, a larger spatial distance value means that the samples have large sentiment differences, and the bimodal mixture contrastive learning model gives less attention to them. A smaller spatial distance value means that they are difficult to distinguish as difficult negative samples, triggering the bimodal mixture contrastive learning model to apply a larger gradient penalty through the dynamic temperature coefficient, so as to force the bimodal mixture contrastive learning model to learn more refined feature discrimination.

[0038] Specifically, with the core objective of obtaining a powerful representation of the original speech signal, the model primarily uses the original speech signal, supplemented by Melanographic spectrogram features and pathological speech signals. A pre-trained model, wav2vec2.0, is used as the encoder, and a two-layer multilayer perceptron (MLP) is employed to construct the projection head. A dual-modal hybrid contrastive loss function is constructed through intra-modal and inter-modal contrastive learning, and optimized using the sentiment PAD metric. The encoder and projection head are then fine-tuned during training. Specifically, the bottom-level network parameters of the wav2vec2.0 pre-trained model are fixed, while only the top-level network parameters and all parameters of the projection head are fine-tuned to balance training effectiveness and efficiency. The bottom-level parameters retain the general feature extraction capabilities obtained during pre-training, while the top-level and projection head parameters are adapted to the dual-modal contrastive learning task to achieve feature optimization in specific scenarios. The logic for contrastive learning and loss function construction is as follows: Intra-modal contrastive learning (IACL) aims to learn the dynamic characteristics within a single modality and the relationships between different emotions in a supervised manner, making the feature space more discriminative. Its implementation logic is as follows: Within a training batch of size K, for each modality in the original speech signal modality, the Mel spectrogram feature modality, and the pathological speech signal modality, positive and negative sample pairs are constructed: Positive sample pair: The anchor sample consists of an anchor sample and M other samples with the same discrete emotion label in the same modality; negative sample pair: The anchor sample consists of an anchor sample and N samples with different discrete emotion labels in the same modality; where N + M = K - 1, and the training batch size K must be set to ensure that the number of samples M ≥ 1 of the same modality and the same discrete emotion label within each batch, to ensure the effective implementation of intra-modal contrastive learning.

[0039] The formula for intramodal contrast loss is as follows: ; In the formula, , representing the original speech signal mel spectrogram and pathological speech signals , , , Voice characteristics of normal people , and pathological speech features The feature vectors after passing through the projection head represent anchor points, positive samples, and negative samples, respectively. The dot product operation is a measure of the similarity between two samples. Before the dot product operation, the representation needs to be L2 regularized to ensure that the similarity between the pairs is between 0 and 1. Here, is the temperature coefficient. Therefore, the intra-modal contrast loss in this embodiment can be expressed as: ; Therefore, by minimizing This allows the model to delve deeper into and extract subtle features and key information within a modality, providing a solid foundation for subsequent cross-modal learning. It also enables more natural interaction and fusion with other modalities, resulting in more accurate multimodal task processing.

[0040] The core objective of Inter-modal Contrastive Learning (IECL) is to achieve effective fusion and interaction between data from different modalities, overcoming the limitations of information within a single modality. Its implementation logic is largely similar to intra-modal contrastive learning, but the definition of positive and negative sample pairs differs: Within a training batch of size K, positive and negative sample pairs are constructed for combinations of the original speech signal modality, Mel spectrogram feature modality, and pathological speech signal modality. Positive sample pairs consist of anchor samples and samples with the same discrete emotion label across different modalities; negative sample pairs consist of anchor samples and samples with different discrete emotion labels across different modalities. Intramodal contrastive learning only considers the relationships between data within the same modality, aiming to make samples of the same emotion within the same modality have similar representations and increase the differentiation between samples of different emotions. However, unimodal models focus only on a specific data modality and cannot adapt well to changes in different scenarios or tasks. When faced with new and unseen data, especially when this data comes from different modalities or formats, the model's performance may be severely affected. The main purpose of intermodal contrastive learning is to achieve effective fusion and interaction between data from different modalities, overcoming the information limitations of unimodal models. Through contrastive learning, the model can learn the similarities and differences between data from different modalities, thereby finding their correlations and complementarities.

[0041] Inter-modal Contrastive Learning (IECL) is largely similar to IACL, but differs in that it aims to learn dynamic characteristics between modalities through contrastive learning. Furthermore, IECL modifies the positive and negative samples used during learning. Positive sample pairs are defined as samples from two different modalities under the same sentiment label; this includes not only all similar samples within the same modality (as in IACL) but also similar samples from different modalities. Negative sample pairs are defined as all samples from two different modalities. Therefore, for anchors in a mini-batch of size K, IECL produces twice the number of positive and negative sample pairs compared to IACL. The loss of IACL can be obtained using the same approach: ; The parameters in the formula are similar to those in IACL. Therefore, the intermodal contrast loss in this embodiment can be expressed as: ; The final multimodal mixture contrast loss (MMCL) is composed of IACL and IECL, and its formula is as follows: ; In the formula, λ1, λ2, and λ3 are weight coefficients that satisfy λ1+λ2+λ3=1. Their values ​​are determined by cross-validation. The range of values ​​for λ1, λ2, and λ3 is 0.1-0.6, which is used to balance the importance of intra-modal and inter-modal contrastive learning.

[0042] Because the training dataset contains similar emotions such as happiness and joy, anger and indifference, sadness and grief, these similar emotions belong to two strong and weak states of the same primary emotion, and are very close in both discrete emotion expression and the dimensional PAD emotion space. In contrastive training, using a fixed temperature coefficient τ has limitations: if the temperature coefficient is too large, the model will not pay enough attention to difficult negative samples, such as negative samples of similar emotions; if the temperature coefficient is too small, the model will over-focus on difficult negative samples, which may lead to a decrease in generalization ability. The dimensional space PAD can accurately measure the similarity between samples: similar emotion samples are closer in the PAD space, while emotionally different samples (such as happiness and anger) are farther apart. Based on this, this application's embodiment introduces PAD into contrastive learning, assigning a real-time temperature coefficient to each pair of positive and negative samples based on the PAD space distance of each sample pair, achieving precise one-to-one punishment.

[0043] Since the embodiments of this application use STEM-E 2 The VA dataset contains similar emotion pairs (happiness and joy), (anger and indifference), and (sadness and grief). These similar emotions represent two strong and weak states of the same primary emotion, and they are quite close in both discrete emotion expression and the PAD emotion space. In contrastive training, the dot product of two samples represents their similarity; the more similar the samples, the closer the similarity is to 1, and vice versa. For example, within a training batch, if a sample is labeled "happiness," then the positive samples are all samples labeled "happiness" within the batch, while the negative samples are samples with other emotion labels. However, due to the subjective differences in emotion expression, there may be pseudo-positive samples with low similarity in positive sample pairs. These samples, although having the same label, have low similarity. Similarly, the similarity characteristic of this database can also lead to pseudo-negative samples with high similarity in negative sample pairs. For example, a sample labeled "happiness" and a sample labeled "joy" may have a dot product similarity close to 1, but are actually negative samples. This increases the learning ability of supervised contrastive learning on difficult positive and negative samples. The temperature coefficient in the contrastive loss function... The temperature coefficient determines the degree of attention that contrastive loss pays to difficult negative samples. A larger temperature coefficient tends to treat all samples equally and will not pay too much attention to more difficult negative samples. On the other hand, a smaller temperature coefficient pays more attention to difficult negative samples that are very similar to the sample, giving difficult negative samples a larger gradient to separate them from positive samples, thus making it easier to form a uniform representation space.

[0044] Temperature coefficient, as a fixed hyperparameter, requires reasonable trade-offs, as an appropriate temperature coefficient can play a decisive role in the model's learning ability. However, as mentioned earlier, both excessively large and excessively small temperature coefficients have limitations. Based on this characteristic, we found that the PAD dimensional space can effectively measure the similarity between samples. For example, for similar emotions like happiness and pleasure, a small temperature coefficient is needed for gradient penalty when calculating the loss. In the PAD dimensional space, the more similar the samples, the closer their spatial distance, allowing for a one-to-one similarity measurement between each pair of samples using PAD. Similarly, for similar sample pairs like happiness and anger, a small temperature coefficient is not necessary, and the PAD dimensional space values ​​can effectively distinguish them. Therefore, we introduce PAD into contrastive learning, assigning a reasonable temperature coefficient to each pair of positive and negative samples based on the PAD spatial distance between each sample, achieving a one-to-one real-time penalty. The PAD spatial distance (Euclidean distance in the PAD 3D emotion space) between each pair of samples within the training batch is calculated using the following formula: ; In the formula, P1, A1, D1 and P2, A2, D2 are the three-dimensional PAD values ​​of two samples, respectively. The formula can be used to obtain a PAD distance matrix of size K×K (K is the training batch size), and the diagonal elements of the matrix are the spatial distances of the samples themselves (with a value of 0).

[0045] Within a training batch K, a PAD distance matrix of size K² is generated between each pair of samples, where the diagonal elements are samples with a spatial distance of 0. The temperature coefficient in the original modal contrastive loss IACL is then applied. Replace with PAD space distance The improved intramodal contrast loss is then expressed as: ; Similarly, the contrast loss of IECL is also replaced in the same way. The final multimodal hybrid contrastive learning loss (MMCL-PAD) with PAD is composed of the improved IACL and IECL, and is expressed as: ; In the formula, λ1, λ2, and λ3 are weight coefficients that satisfy λ1+λ2+λ3=1. Their values ​​are determined by cross-validation. The range of values ​​for λ1, λ2, and λ3 is 0.1-0.6, which is used to balance the importance of intra-modal and inter-modal contrastive learning.

[0046] Step 103: Train downstream speech recognition tasks based on features extracted by the dual-modal hybrid contrastive learning model.

[0047] The downstream speech recognition task training in this application embodiment includes at least a speech emotion recognition task and an automatic speech recognition task.

[0048] This step uses the SenseVoice-Small model for training, applying the powerful feature representations learned during the contrastive learning training phase to specific speech tasks, achieving functions such as speech recognition and emotion recognition. The specific implementation logic is as follows: First, construct the encoder input features: take the feature vector F output from the contrastive learning training phase... feat The input features X of the encoder are concatenated with four special task embeddings. in ,satisfy: X in =[E task E style E lang E speaker ; F feat ]; Among them, E task E is the task embedding vector, used to specify the current training task as speech recognition, emotion recognition, etc. style E embeds style vectors to adapt to different speech styles. lang E is a language embedding vector used to adapt to different languages. speaker Speaker embedding vectors are used to distinguish different speakers. During the input feature construction process, the feature vectors also need to be processed by layer normalization to make the mean of the feature vectors 0 and the variance 1, so as to improve the stability of model training.

[0049] Then, input feature X in The input is fed into a Memory Equipped Self-Attention (SAN-M) encoder for training. During training, a dual loss function is introduced: on the one hand, the multimodal hybrid contrastive loss (MMCL-PAD) of the contrastive learning stage is retained, and on the other hand, the dimensional sentiment MSE loss function is added to jointly optimize the model parameters.

[0050] The calculation logic of the dimensional sentiment MSE loss function is as follows: taking the predicted dimensional sentiment PAD values ​​output by the model and the true PAD values ​​labeled in the training data as inputs, the mean squared error of the two is calculated, and the formula is: ; In the formula, P pred A pred D pred P represents the predicted PAD value output by the model. true A true Dtrue The PAD values ​​labeled in the training data are used as the loss function to enable the model to accurately learn dimensional sentiment features, and to achieve fine-grained sentiment recognition by combining discrete sentiment labels.

[0051] After the model is trained, during the speech recognition inference process, there is no need to manually process the input speech signal data. The original speech signal data to be recognized is directly input into the trained model. The model outputs speech recognition results, discrete emotion recognition results and dimensional emotion PAD index through internal feature extraction, contrast enhancement and decoding, realizing end-to-end speech recognition and fine-grained emotion analysis.

[0052] Figure 2 This is a schematic diagram of the composition structure of the speech recognition framework based on dual-modal hybrid contrast enhancement provided in the embodiments of this application, as shown below. Figure 2 As shown, the dual-modal hybrid contrastive fusion framework of this application mainly includes two stages: supervised contrastive hybrid training and multiple downstream speech training tasks. In the entire model, this application aims to obtain a strong representation of the original speech signal; therefore, the design primarily uses the original speech signal, with the other two modalities supplementing the information. The model is divided into two training stages. In the contrastive learning training stage, the encoder uses the pre-trained model wav2vec2.0, and the projection head consists of a two-layer MLP. To adapt to contrastive learning, this application adopts a partial fine-tuning method for training. In the downstream task stage, the SenseVoice-small training method is used, embedding different tasks as different embeddings into the features. This data is then input into the sequence model. See the previous section for details of the processing. Figure 1 The processing steps shown will not be described in detail here.

[0053] Figure 3 This is a schematic diagram of the composition structure of the speech recognition device based on dual-modal hybrid contrast enhancement provided in the embodiments of this application, as shown below. Figure 3 As shown, the speech recognition device based on dual-modal hybrid contrast enhancement according to an embodiment of this application includes: The acquisition unit 30 is used to acquire multimodal speech training data; the multimodal speech training data includes at least the original speech signal data and the Mel speech spectrogram data generated from the original speech signal data; Construction unit 31 is used to construct and train a bimodal hybrid contrastive learning model. Specifically, samples with the same sentiment label within the same modality are treated as positive sample pairs, and samples with different sentiment labels are treated as negative sample pairs. Intramodal contrastive loss is calculated to enhance the bimodal hybrid contrastive learning model's ability to distinguish fine-grained sentiment features within a single modality. Furthermore, samples with the same sentiment label in different modalities are treated as positive sample pairs, and samples with different sentiment labels in different modalities are treated as negative sample pairs. Intermodal contrastive loss is calculated to achieve alignment and complementarity of feature spaces across different modalities. The intramodal contrastive loss and the intermodal contrastive loss are fused to obtain a multimodal hybrid contrastive loss, and the model parameters of the bimodal hybrid contrastive learning model are optimized. Training unit 32 is used to train downstream speech recognition tasks based on features extracted by the dual-modal hybrid contrastive learning model; the training of downstream speech recognition tasks includes at least speech emotion recognition tasks and automatic speech recognition tasks.

[0054] In some embodiments, the multimodal speech training data further includes pathological speech signal data, which is speech data of patients with dysarthria. In the intramodal and intermodal contrastive learning of the dual-modal hybrid contrastive learning model, the pathological speech signal data is treated as an independent modality and participates in the contrastive learning training together with the original speech signal data and the Mel spectrogram data, so that the normal speech feature space of the dual-modal hybrid contrastive learning model can learn and approximate the pathological speech features.

[0055] In some implementations, the speech emotion recognition task includes discrete emotion recognition tasks and dimensional emotion recognition tasks; The samples in the multimodal speech training data are all labeled with dimensional sentiment labels, which are the coordinate values ​​of the PAD in the three-dimensional sentiment space; In the contrastive learning training of the bimodal hybrid contrastive learning model, the contrastive loss function is dynamically adjusted using the aforementioned dimensional sentiment labels.

[0056] In some implementations, the construction unit 31 dynamically adjusts the contrastive loss function using the dimensional sentiment labels, including: For any two training samples in the training batch of the bimodal hybrid contrastive learning model, calculate their spatial distance in the PAD three-dimensional emotion space; When calculating intramodal contrast loss or intermodal contrast loss, the fixed temperature coefficient in the contrast loss function is replaced with a dynamic temperature coefficient associated with the training sample pair. Specifically, for positive sample pairs, a smaller spatial distance value means that the training samples are highly similar in the sentiment dimension, and the bimodal mixture contrastive learning model pays more attention to them to bring their feature representations closer together. For negative sample pairs, a larger spatial distance value means that the samples have large sentiment differences, and the bimodal mixture contrastive learning model pays less attention to them. On the other hand, a smaller spatial distance value means that they are difficult negative samples that are hard to distinguish, which triggers the bimodal mixture contrastive learning model to apply a larger gradient penalty through a dynamic temperature coefficient, so as to force the bimodal mixture contrastive learning model to learn more refined feature discrimination.

[0057] In some implementations, the encoder of the dual-modal hybrid contrastive learning model employs a pre-trained self-supervised learning speech representation model wav2vec, and connects a projection head composed of two multilayer perceptrons after the output layer of wav2vec to map the encoded features to the low-dimensional representation space required for contrastive learning; during training, a partial adjustment strategy is adopted to update the parameters of the projection head and adjust some high-level network parameters of the encoder.

[0058] In some implementations, the downstream speech recognition task is trained using a non-autoregressive SenseVoice-small model with an encoder-only structure; The fusion features extracted by the trained bimodal hybrid contrastive learning model are concatenated with identity embedding vectors representing different tasks to form an enhanced input feature sequence. The enhanced input feature sequence is then input into the encoder of the SenseVoice-small model for multi-task joint training. The task identity embedding vectors include at least automatic speech recognition embedding and speech emotion recognition embedding.

[0059] In some implementations, during the training of downstream speech recognition tasks, the loss function of the training unit 32 for speech emotion recognition tasks is composed of the cross-entropy loss of discrete emotion classification and the mean squared error loss of dimensional emotion prediction, so as to achieve joint modeling of discrete categories and continuous dimensional attributes of emotion.

[0060] In an exemplary embodiment, each processing module in the speech recognition device based on dual-modal hybrid contrast enhancement of this application embodiment can be implemented by one or more central processing units (CPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components.

[0061] Regarding the apparatus in the above embodiments, the specific manner in which each module and unit performs its operations has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0062] Figure 4 This is a schematic structural diagram of an electronic device 400 provided in an embodiment of this application. The electronic device may be a personal computer or a server. Figure 4 The illustrated electronic device 400 includes a processor 410, which can call and run computer programs from memory to implement the methods in the embodiments of this application.

[0063] Optionally, such as Figure 4 As shown, the electronic device 400 may further include a memory 420. The processor 410 can retrieve and run computer programs from the memory 420 to implement the methods described in the embodiments of this application.

[0064] The memory 420 can be a separate device independent of the processor 410, or it can be integrated into the processor 410.

[0065] Optionally, such as Figure 4 As shown, the electronic device 400 may also include a transceiver 430, which the processor 410 can control to communicate with other devices. Specifically, it can send information or data to other devices or receive information or data sent by other devices.

[0066] The transceiver 430 may include a transmitter and a receiver. The transceiver 430 may further include an antenna, and the number of antennas may be one or more.

[0067] The electronic device 400 can implement the corresponding processes of the various methods in the embodiments of this application, which will not be described in detail here for the sake of brevity. That is to say, the electronic device in the embodiments of this application can be implemented as a server or as a client. For the sake of simplicity, they will not be presented separately. The main difference lies in the different methods corresponding to the programs they run, while their hardware structures are exactly the same.

[0068] Figure 5 This is a schematic structural diagram of the chip according to an embodiment of this application. Figure 5 The chip 500 shown includes a processor 510, which can call and run computer programs from memory to implement the methods in the embodiments of this application.

[0069] Optionally, such as Figure 5 As shown, chip 500 may further include memory 520. Processor 510 can retrieve and run computer programs from memory 520 to implement the methods described in this embodiment.

[0070] The memory 520 can be a separate device independent of the processor 510, or it can be integrated into the processor 510.

[0071] Optionally, the chip 500 may also include an input interface 530. The processor 510 can control the input interface 530 to communicate with other devices or chips; specifically, it can acquire information or data sent by other devices or chips.

[0072] Optionally, the chip 500 may also include an output interface 540. The processor 510 can control the output interface 540 to communicate with other devices or chips, specifically, to output information or data to other devices or chips.

[0073] This chip can implement the corresponding processes of the various methods implemented in the embodiments of this application, which will not be described in detail here for the sake of brevity.

[0074] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0075] It should be understood that the processor in the embodiments of this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. The functionality of a chip can be implemented using different processing methods depending on the program running in its memory.

[0076] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0077] It should be understood that the above-described memory is exemplary and not a limiting description. For example, the memory in the embodiments of this application may also be static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DR RAM), etc. That is to say, the memory in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0078] This application also provides a computer-readable storage medium for storing a computer program. This computer program causes a computer to execute the corresponding processes implemented by the various methods of this application embodiment; for brevity, these will not be elaborated upon here.

[0079] This application also provides a computer program product, including computer program instructions. These computer program instructions cause a computer to execute the corresponding processes implemented by the various methods of this application embodiment; for brevity, they will not be described in detail here.

[0080] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0081] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0082] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0083] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0084] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0085] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0086] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A speech recognition method based on dual-modal hybrid contrast enhancement, characterized in that, The method includes: Acquire multimodal speech training data; the multimodal speech training data includes at least the original speech signal data and the Mel speech spectrogram data generated from the original speech signal data; A bimodal hybrid contrastive learning model is constructed and trained. Samples with the same sentiment label within the same modality are treated as positive sample pairs, and samples with different sentiment labels are treated as negative sample pairs. Intramodal contrastive loss is calculated to enhance the model's ability to distinguish fine-grained sentiment features within a single modality. Furthermore, samples with the same sentiment label in different modalities are treated as positive sample pairs, and samples with different sentiment labels in different modalities are treated as negative sample pairs. Intermodal contrastive loss is calculated to achieve alignment and complementarity of feature spaces across different modalities. The intramodal and intermodal contrastive losses are then fused to obtain a multimodal hybrid contrastive loss, and the model parameters of the bimodal hybrid contrastive learning model are optimized. Based on the features extracted by the dual-modal hybrid contrastive learning model, downstream speech recognition tasks are trained, which include at least speech emotion recognition tasks and automatic speech recognition tasks.

2. The method according to claim 1, characterized in that, The multimodal speech training data also includes pathological speech signal data, which is the speech data of patients with dysarthria. In the intramodal and intermodal contrastive learning of the dual-modal hybrid contrastive learning model, the pathological speech signal data is treated as an independent modality and participates in the contrastive learning training together with the original speech signal data and the Mel spectrogram data, so that the normal speech feature space of the dual-modal hybrid contrastive learning model can learn and approximate the pathological speech features.

3. The method according to claim 1, characterized in that, The speech emotion recognition task includes discrete emotion recognition task and dimensional emotion recognition task; The samples in the multimodal speech training data are all labeled with dimensional sentiment tags, which are the coordinate values ​​of pleasure, activation and dominance PAD in the three-dimensional sentiment space. In the contrastive learning training of the bimodal hybrid contrastive learning model, the contrastive loss function is dynamically adjusted using the aforementioned dimensional sentiment labels.

4. The method according to claim 3, characterized in that, The method of dynamically adjusting the contrast loss function using the aforementioned dimensional sentiment tags includes: For any two training samples in the training batch of the bimodal hybrid contrastive learning model, calculate their spatial distance in the PAD three-dimensional emotion space; When calculating intramodal contrast loss or intermodal contrast loss, the fixed temperature coefficient in the contrast loss function is replaced with a dynamic temperature coefficient associated with the training sample pair. Specifically, for positive sample pairs, a smaller spatial distance value means that the training samples are highly similar in the sentiment dimension, and the bimodal mixture contrastive learning model pays more attention to them to bring their feature representations closer together. For negative sample pairs, a larger spatial distance value means that the samples have large sentiment differences, and the bimodal mixture contrastive learning model pays less attention to them. On the other hand, a smaller spatial distance value means that they are difficult negative samples that are hard to distinguish, which triggers the bimodal mixture contrastive learning model to apply a larger gradient penalty through a dynamic temperature coefficient, so as to force the bimodal mixture contrastive learning model to learn more refined feature discrimination.

5. The method according to any one of claims 1 to 4, characterized in that, The encoder of the dual-modal hybrid contrastive learning model uses a pre-trained self-supervised learning speech representation model wav2vec, and connects a projection head composed of two multilayer perceptrons after the output layer of wav2vec to map the encoded features to the low-dimensional representation space required for contrastive learning. During training, a partial adjustment strategy is adopted to update the parameters of the projection head and adjust some high-level network parameters of the encoder.

6. The method according to claim 1, characterized in that, The downstream speech recognition task training uses a non-autoregressive SenseVoice-small model with an encoder-only structure; The fusion features extracted by the trained bimodal hybrid contrastive learning model are concatenated with identity embedding vectors representing different tasks to form an enhanced input feature sequence. The enhanced input feature sequence is then input into the encoder of the SenseVoice-small model for multi-task joint training. The task identity embedding vectors include at least automatic speech recognition embedding and speech emotion recognition embedding.

7. The method according to claim 1, characterized in that, In the training of downstream speech recognition tasks, for speech emotion recognition tasks, the loss function is composed of the cross-entropy loss of discrete emotion classification and the mean squared error loss of dimensional emotion prediction, so as to achieve joint modeling of discrete categories and continuous dimensional attributes of emotion.

8. An electronic device, characterized in that, include: A processor and a memory for storing a computer program, the processor for calling and running the computer program stored in the memory to perform the method as described in any one of claims 1 to 7.

9. A chip, comprising: A processor for retrieving and running a computer program from memory, causing a device with the chip mounted to perform a speech recognition method based on bimodal hybrid contrast enhancement as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, It includes computer program instructions that cause a computer to perform the speech recognition method based on bimodal hybrid contrast enhancement as described in any one of claims 1 to 7.

Images