Emotion recognition method and device based on voice large model, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses an emotion recognition method and device based on a voice large model, equipment and a medium, which are applied to a patient emotion recognition scene in the medical field, and the method comprises the steps: obtaining voice segment information and text description information corresponding to the voice segment information; performing feature coding and feature alignment on the voice fragment information and the text description information to generate joint features, and performing self-supervised training on the pre-trained voice large model through the joint features to generate an initial voice large model; collecting a voice instruction pair, and performing model fine tuning on the initial voice large model to obtain a fine-tuned voice large model; obtaining voice information with diversified emotion tags, optimizing the target function of the fine-tuned voice large model, and generating a target emotion recognition model; and obtaining to-be-recognized voice segment information, and performing emotion recognition based on the target emotion recognition model to obtain an emotion recognition result. The accuracy of emotion recognition is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and particularly to an emotion recognition method, device, equipment and medium based on a speech large model. Background Art

[0002] Emotion recognition technology is of great significance in the fields of fintech and healthcare, such as being applied to the scenario of patient emotion recognition in the healthcare field or the scenario of recognizing customer emotions in the financial field. However, the existing technologies still face many technical bottlenecks in practical applications: the recognition accuracy drops significantly in a high background noise environment, resulting in misjudgment of users' angry emotions in the intelligent customer service scenario.

[0003] The existing methods have limitations in feature expression ability, making it difficult to effectively capture the subtle emotion features in speech signals; in multi-emotion classification tasks, traditional models are difficult to distinguish similar emotion states; in cross-domain applications, the model generalization performance is insufficient; environmental noise interference seriously affects the recognition robustness. The current solutions mostly adopt the method of cascading multiple models, such as first performing audio noise reduction processing and then performing emotion recognition, or improving the generalization ability through pre-adaptation technology, but these methods have problems such as high system complexity, error accumulation, and large consumption of computing resources. Especially when dealing with complex scenarios, traditional methods are difficult to achieve end-to-end robust emotion recognition. With the breakthrough of large model technology in the field of text processing, how to introduce the advantages of large models into the field of speech emotion recognition, build an emotion recognition system with strong feature learning ability and environmental adaptability, and improve the accuracy of emotion recognition has become a technical problem to be solved urgently. Summary of the Invention

[0004] The purpose of the embodiments of this application is to propose an emotion recognition method, device, equipment and medium based on a speech large model to improve the accuracy of emotion recognition.

[0005] To solve the above technical problems, the embodiments of this application provide an emotion recognition method based on a speech large model, including:

[0006] Obtain speech segment information and the corresponding text description information of the speech segment information;

[0007] Perform feature encoding and feature alignment on the speech segment information and the text description information to generate joint features, and perform self-supervised training on a pre-trained speech large model through the joint features to generate an initial speech large model;

[0008] Collect speech instruction pairs, and perform model fine-tuning on the initial speech large model based on the speech instruction pairs to obtain a fine-tuned speech large model;

[0009] Obtain voice information with diverse emotion tags, and optimize the objective function of the fine-tuned large voice model based on the voice information to generate a target emotion recognition model;

[0010] Obtain the voice segment information to be recognized, and perform emotion recognition on the voice segment information to be recognized based on the target emotion recognition model to obtain an emotion recognition result.

[0011] To solve the above technical problems, an embodiment of the present application provides an emotion recognition device based on a large voice model, including:

[0012] An initial information acquisition module, configured to acquire voice segment information and text description information corresponding to the voice segment information;

[0013] A feature alignment module, configured to perform feature encoding and feature alignment on the voice segment information and the text description information to generate a joint feature, and perform self-supervised training on a pre-trained large voice model through the joint feature to generate an initial voice model;

[0014] A model fine-tuning module, configured to collect voice instruction pairs, and perform model fine-tuning on the initial voice model based on the voice instruction pairs to obtain a fine-tuned large voice model;

[0015] An objective function optimization module, configured to obtain voice information with diverse emotion tags, and optimize the objective function of the fine-tuned large voice model based on the voice information to generate a target emotion recognition model;

[0016] An emotion recognition module, configured to obtain the voice segment information to be recognized, and perform emotion recognition on the voice segment information to be recognized based on the target emotion recognition model to obtain an emotion recognition result.

[0017] To solve the above technical problems, a technical solution adopted by the present invention is: to provide a computer device, including one or more processors; a memory, configured to store one or more programs, so that one or more processors implement the emotion recognition method based on a large voice model described in any one of the above.

[0018] To solve the above technical problems, a technical solution adopted by the present invention is: a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the emotion recognition method based on a large voice model described in any one of the above is implemented.

[0019] An embodiment of the present invention provides a method, apparatus, device, and medium for emotion recognition based on a speech large model. The method includes: obtaining speech segment information and corresponding text description information of the speech segment information; performing feature encoding and feature alignment on the speech segment information and the text description information to generate a joint feature, and performing self-supervised training on a pre-trained speech large model through the joint feature to generate an initial speech large model; collecting speech instruction pairs, and performing model fine-tuning on the initial speech large model based on the speech instruction pairs to obtain a fine-tuned speech large model; obtaining speech information with diverse emotion labels, and optimizing an objective function of the fine-tuned speech large model based on the speech information to generate a target emotion recognition model; obtaining speech segment information to be recognized, and performing emotion recognition on the speech segment information to be recognized based on the target emotion recognition model to obtain an emotion recognition result. The embodiment of the present invention solves the problems of low recognition accuracy and insufficient feature expression ability of traditional methods in a noisy environment by training a large model with the joint feature of fusing speech segments and text description information, and adopting a multi-stage optimization strategy, and has the advantages of improving emotion recognition accuracy, robustness, and enhancing the generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the solutions in the present application, the following will briefly introduce the drawings required for the description of the embodiments of the present application. Obviously, the drawings in the following description are some embodiments of the present application, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

[0021] Figure 1 is a schematic diagram of an application environment of a method for emotion recognition based on a speech large model in an embodiment of the present invention;

[0022] Figure 2 is a flowchart of the implementation of a method for emotion recognition based on a speech large model provided by an embodiment of the present application;

[0023] Figure 3 is Figure 2 a schematic flowchart of a specific implementation manner of step S2 in

[0024] Figure 4 is Figure 3 a schematic flowchart of a specific implementation manner of step S23 in

[0025] Figure 5 is Figure 2 a schematic flowchart of a specific implementation manner of step S3 in

[0026] Figure 6 is Figure 2 a schematic flowchart of a specific implementation manner of step S4 in

[0027] Figure 7 is Figure 2 A schematic flow chart of a specific implementation manner of step S4 in

[0028] Figure 8 A schematic diagram of an emotion recognition device based on a speech large model provided by an embodiment of the present application;

[0029] Figure 9 A schematic diagram of a computer device provided by an embodiment of the present application. Specific implementation manner

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs; the terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above drawings are intended to cover non-exclusive inclusion. The terms "first", "second", etc. in the specification and claims of this application or the above drawings are used to distinguish different objects and not to describe a specific order.

[0031] Referring to "embodiment" herein means that a specific feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of this application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0032] In order to enable those skilled in the art to better understand the solution of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the drawings.

[0033] The present invention will be described in detail below with reference to the drawings and embodiments.

[0034] It should be noted that the emotion recognition method based on a speech large model provided by the embodiments of this application is generally executed by a server. Correspondingly, the emotion recognition device based on a speech large model is generally configured in the server.

[0035] The emotion recognition method based on a speech large model provided by the embodiments of the present invention can be applied in, for example Figure 1In the application environment, the client communicates with the server through the network. The server can receive the voice segment information of the client; generate an emotion recognition result according to the voice segment information. In the server of the present invention, the emotion recognition result is sent to the client. Among them, the client can be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers. The present invention will be described in detail through specific embodiments below.

[0036] The emotion recognition method based on the speech large model provided by the embodiments of the present application can be applied to the emotion recognition scenario of patients in the medical field, or applied to the scenario of recognizing customer emotions in the financial field.

[0037] Please refer to Figure 2 , Figure 2 which shows a specific implementation manner of the emotion recognition method based on the speech large model.

[0038] It should be noted that if there are substantially the same results, the method of the present invention is not limited to Figure 2 the process sequence shown, and the method includes the following steps:

[0039] S1: Obtain the voice segment information and the text description information corresponding to the voice segment information.

[0040] Specifically, the voice segment information refers to the original audio data containing emotional features. Specifically, the spectrogram after frame division and windowing can be used as the input, and the acoustic features are extracted through the Mel Frequency Cepstral Coefficients to capture emotion-related parameters such as pitch and speech rate. The text description information refers to the semantic content corresponding to the voice segment. Specifically, the text transcription can be generated through an automatic speech recognition system, or key emotional keywords can be manually marked to provide the semantic context for emotion classification.

[0041] Furthermore, in a specific embodiment, the voice segment information is denoised, and the text description information is preprocessed such as text completion and text cleaning. Preprocessing the voice segment information and the text description information is beneficial to improving the accuracy of subsequent feature extraction, and thus beneficial to improving the accuracy of emotion recognition.

[0042] S2: Perform feature encoding and feature alignment on the voice segment information and the text description information to generate a joint feature, and perform self-supervised training on the pre-trained speech large model through the joint feature to generate an initial speech large model.

[0043] Specifically, the voice segment information is converted into voice features by a voice encoder, and the text description information is converted into text embeddings by an embedding layer. Subsequently, through a linear projection layer, the voice features and text embeddings are aligned with each other. Then, the positions of the audio embeddings in the text are integrated to form joint features. Then, the joint features are input into a pre-trained large language model to output the next text embedding to be predicted. A self-supervised training dataset is constructed, and the pre-trained voice large model is trained through cross-entropy loss to generate an initial voice large model.

[0044] Please refer to Figure 3 , Figure 3 which shows a specific implementation manner of step S2, described in detail as follows:

[0045] S21: Convert the voice segment information into a high-dimensional feature vector through a voice encoder.

[0046] S22: Convert the text description information into a text embedding vector through an embedding layer.

[0047] S23: Align and fuse the high-dimensional feature vector and the text embedding vector to generate the joint feature.

[0048] S24: Use the cross-entropy loss function to perform self-supervised training on the pre-trained voice large model based on the joint feature to generate the initial voice large model.

[0049] Among them, the voice encoder refers to a neural network module that converts voice signals into high-dimensional feature vectors, and can be specifically implemented by a convolutional neural network or a Transformer structure, and is used to extract acoustic features such as rhythm and pitch in the voice. The embedding layer refers to a conversion layer that maps text descriptions into vector representations, and can be specifically implemented by a pre-trained word vector model or a BERT structure, and is used to capture semantic information in the text. Feature alignment refers to an operation to eliminate the distribution differences between features of different modalities, and can be specifically achieved by mapping voice and text features to the same latent space through a linear projection layer, so that features of different modalities are comparable. The cross-entropy loss function refers to a mathematical function that measures the difference between the model prediction and the true label, and can be specifically used to optimize model parameters and drive the model to learn the joint distribution pattern of bimodal features.

[0050] Specifically, after the speech encoder frames and windows the input speech, it extracts time-frequency domain features through multi-layer convolution operations, and finally generates a high-dimensional feature vector with temporal characteristics. The embedding layer tokenizes the text description and generates a context-related text embedding vector through look-up table or encoder. The linear projection layer maps the vectors of the two modalities to a latent space of the same dimension, calculates the matching degree between the modalities using cosine similarity, and adjusts the parameters of the projection matrix through backpropagation to achieve feature alignment. The fused joint features are input into the pre-trained large speech model, and the difference between the current prediction and the self-supervised generated label is calculated through cross-entropy loss, prompting the model to establish an associative mapping between speech features and text semantics, thereby enhancing the model's collaborative understanding ability of multi-modal information in a noisy interference scenario.

[0051] Traditional methods usually adopt a concatenated structure that independently processes speech and text features, without establishing a deep association mechanism between modalities, resulting in insufficient utilization of cross-modal information. In contrast, the embodiments of the present application achieve the alignment of bimodal features through latent space mapping, and optimize the model parameters using self-supervised training of joint features, effectively solving the modality gap problem caused by the difference in feature distributions and improving the robustness of the model in complex acoustic environments. The present application can achieve precise alignment between speech features and text semantics in high background noise scenarios. For example, in an intelligent customer service system, when the acoustic features of the user's speech are blurred due to environmental noise, the system can accurately judge the user's emotional state by associating with the text keywords in the call record. This solution effectively avoids the problem of misjudging emotions caused by mismatched modality features in traditional methods and improves the reliability of collaborative analysis of multi-source information.

[0052] Please refer to Figure 4 , Figure 4 which shows a specific implementation of step S23, described in detail as follows:

[0053] S231: Map the high-dimensional feature vector and the text embedding vector to the same latent space through the linear projection layer to align the high-dimensional feature vector and the text embedding vector, obtaining the aligned high-dimensional feature vector and the aligned text embedding vector.

[0054] S232: Integrate the position encoding of the aligned high-dimensional feature vector with the position information of the aligned text embedding vector to generate the joint feature.

[0055] Among them, the linear projection layer refers to a learnable parameter matrix that maps vectors of different modalities to the same dimension. Specifically, it can be implemented using a fully connected neural network. By adjusting the weight parameters, the speech and text features can have a consistent distribution in the latent space. The latent space refers to a unified vector space after the transformation of data of different modalities. Specifically, it can adopt the structure of Euclidean space or manifold space to enable a computable distance metric between the speech spectral features and the text semantic features. The positional encoding refers to an embedding vector that represents the temporal position of the speech signal. Specifically, it can adopt a fixed encoding generated by a sine function or a trainable positional embedding layer to retain the time series characteristics of the speech segment. The positional information integration refers to an operation that associates the word order position of the text with the speech frame position. Specifically, it can adopt the positional bias term or the gated fusion module in the attention mechanism to enable the joint feature to carry both temporal dynamic changes and semantic static expressions.

[0056] Specifically, the high-dimensional feature vector output by the speech encoder is transformed by the linear projection layer to form a vector representation with the same dimension as the text embedding vector, eliminating the spatial distribution difference of cross-modal data. The projected speech features and text features are calibrated in distribution through cosine similarity calculation in the latent space, enabling the fundamental frequency change features of the speech to form a corresponding relationship with the emotional keywords of the text in the vector space. In the feature fusion stage, the temporal positional encoding carried by the speech features is superimposed on the position matrix of the text features, and the positional association between the frame-level speech segments and the word-level text segments is established through the multi-head attention mechanism to form a joint feature vector that simultaneously contains acoustic temporal characteristics and semantic context characteristics. This dual positional association mechanism enables the model to accurately capture the mapping relationship between the emotional fluctuations at specific time points and the corresponding text descriptions when processing long speech segments.

[0057] Traditional cross-modal alignment methods usually optimize the embedding spaces of speech and text separately, resulting in the ineffective elimination of the feature distribution deviation between modalities. Existing technologies only encode a single modality when processing positional information and do not establish cross-modal temporal-semantic positional associations, causing the synergistic effect between speech intonation changes and the appearance of text emotional words to be ignored during emotion recognition. The embodiments of this application overcome the problem of inaccurate feature matching caused by the heterogeneity of multi-modal data by forcing the consistency of the latent space distribution and cross-modal positional associations. The embodiments of this application achieve the precise alignment of speech features and text features in the same space, solving the problem of the loss of emotion-related information caused by modality differences. By fusing dual positional information, the joint modeling ability of the model for the temporal changes of speech signals and the semantic structure of text is enhanced, and the text keywords corresponding to the angry emotion in the user's speech can still be accurately recognized in a noisy environment, effectively improving the accuracy of work order priority determination in the intelligent customer service scenario.

[0058] S3: Collect speech instruction pairs, and fine-tune the initial speech large model based on the speech instruction pairs to obtain a fine-tuned speech large model.

[0059] Specifically, collect voice instruction pairs, and perform full-scale fine-tuning on the parameters of the initial voice large model based on the voice instruction pairs to obtain the initially fine-tuned voice large model; insert a low-rank matrix into the initially fine-tuned voice large model, and use an adapter to insert lightweight adaptation modules into each layer of the initially fine-tuned voice large model to obtain the fine-tuned voice large model.

[0060] In a specific embodiment, the input of the fine-tuned voice large model is: <|voice|>Please determine what the emotion of this voice is? The output of the fine-tuned voice large model is: The emotion of this voice is sad because the voice is particularly low, the speaking speed is relatively slow, and the content of the speech shows that the speaker failed the exam, so it is inferred that the emotion of the speaker is relatively sad.

[0061] Please refer to Figure 5 , Figure 5 which shows a specific implementation manner of step S3, described in detail as follows:

[0062] S31: Collect the voice instruction pairs, and perform full-scale fine-tuning on the parameters of the initial voice large model based on the voice instruction pairs to obtain the initially fine-tuned voice large model.

[0063] S32: Insert a low-rank matrix into the initially fine-tuned voice large model, and use an adapter to insert lightweight adaptation modules into each layer of the initially fine-tuned voice large model to obtain the fine-tuned voice large model.

[0064] Specifically, full-scale fine-tuning refers to the operation of adjusting all trainable parameters of the model, which can be specifically implemented by using an optimization algorithm based on gradient descent. For example, use the Adam optimizer to update all network weights to make the model initially adapt to the feature distribution of specific voice instruction pairs. A low-rank matrix is a mathematical structure that reduces the parameter dimension through matrix decomposition. Specifically, the singular value decomposition method can be used to generate a weight matrix after dimensionality reduction, which is used to compress the parameter space scale after full-scale fine-tuning. An adapter is a computational tool for efficient parameter adjustment, which can be specifically implemented by using low-rank adaptation technology. For example, add trainable bypass branches between Transformer layers through the LoRA method. A lightweight adaptation module is an adjustment component with a parameter quantity much smaller than that of the original model, which can be specifically implemented by using a linear projection layer. For example, connect two fully connected networks after the multi-head attention mechanism for feature transformation.

[0065] Specifically, after the initial speech large model completes full-scale fine-tuning, first perform low-rank decomposition on the weight matrices of each layer of the network. For example, decompose the weight matrix of the fully connected layer into the product form of two low-dimensional matrices, retaining 95% of the singular value energy of the original matrix. Subsequently, adaption modules are added in parallel in each layer of the neural network. This module consists of a linear projection layer and an activation function, and its input and output dimensions are the same as those of the original layer, but the internal parameter scale is only 10% of the original layer. During the training process, freeze the original network parameters and only update the gradients of the low-rank matrices and the adaption modules. Through the hierarchical insertion structure design, under the supervision signal of the speech command pair, the adaption module can perform differential adjustment on the acoustic features extracted at different levels. For example, at the bottom layer network, it focuses on correcting the spectral features under noise interference, and at the high layer network, it optimizes the association mapping of semantics and emotions.

[0066] Traditional methods usually adopt single full-scale fine-tuning or adapter fine-tuning with a fixed backbone. Although full-scale fine-tuning can fully adapt to the target task, it is prone to overfitting of the training data and consumes a huge amount of computing resources; the adapter with a fixed backbone saves resources, but is limited by the small-scale parameters and difficult to capture complex emotion features. This solution uses two-stage progressive optimization. First, use full-scale fine-tuning to establish the basic task adaptation ability, and then achieve precise parameter adjustment through low-rank compression and hierarchical adaptation. While retaining the representation ability of the backbone network, the number of model parameters is reduced by about 65%, and the training speed is increased by 2.3 times. This application effectively alleviates the overfitting phenomenon in the full-scale fine-tuning process. In the actual measurement of the intelligent customer service scenario, the fluctuation range of the emotion recognition accuracy of the model on the test set containing background noise is reduced by 18 percentage points. At the same time, the hierarchical adaption module enables the model to quickly adjust according to different scenario requirements. For example, it automatically enhances the high-frequency feature extraction ability in the in-vehicle environment noise and strengthens the sensitivity to intonation changes in the psychological counseling scenario, achieving stable emotion recognition across scenarios.

[0067] S4: Obtain speech information with diverse emotion labels, and optimize the objective function of the fine-tuned speech large model based on the speech information to generate an objective emotion recognition model.

[0068] Specifically, obtain speech information with diverse emotion labels, use the fine-tuned speech large model to perform recognition based on the speech information to generate a recognition result, calculate the benefit value according to the recognition result and the emotion label, and use the policy gradient algorithm to optimize the objective function based on the benefit value to generate an objective emotion recognition model.

[0069] In a specific embodiment, construct a data set based on the speech information in the following manner: Input: <|Speech|> Please judge whether the emotion of this speech is anger? Output o1: Yes, because the speaker said swear words and is probably trying to express anger; Output o2: No, because the speaker's voice is calm and the speech rate is moderate; Output o3:...; Output oG:....

[0070] Please refer to Figure 6 , Figure 6 which shows a specific implementation of step S4, described in detail as follows:

[0071] S41: Obtain the speech information with diverse emotion labels.

[0072] S42: Based on the speech information, perform recognition through the fine-tuned speech large model to generate a recognition result, and calculate a benefit value according to the recognition result and the emotion labels in the speech information.

[0073] S43: Use the policy gradient algorithm to optimize the objective function based on the benefit value to generate the target emotion recognition model.

[0074] Among them, the diverse emotion labels refer to the labeled data covering multiple emotion categories and intensity levels. Specifically, a multi-level label system including basic emotions such as anger, joy, and sadness and their mixed states can be adopted to enhance the model's ability to distinguish complex emotion features. The benefit value is a quantitative index representing the matching degree between the model's prediction result and the true label. Specifically, a weighted function based on classification accuracy and confusion matrix parameters can be used to calculate it, dynamically reflecting the performance differences of the model in different emotion categories. The policy gradient algorithm is a reinforcement learning method that directly adjusts the model's policy parameters through gradient update. Specifically, the proximal policy optimization algorithm can be adopted to achieve it, guiding the model parameters to iterate in the direction of high return by maximizing the expected benefit value. The optimization of the objective function refers to the process of dynamically adjusting the model parameters in the reinforcement learning framework. Specifically, the KL divergence constraint can be combined to prevent the policy update amplitude from being too large and maintain the stability of model training.

[0075] Specifically, first establish a speech dataset containing multi-dimensional emotion labels to cover the emotion expression differences in different scenarios. In the model recognition stage, input the speech information into the fine-tuned speech large model to generate a recognition result in the form of a probability distribution. Based on the comparison between the prediction result and the true label, construct a benefit function including classification accuracy and class discrimination. For example, set a higher penalty weight for the recognition errors in high-confusion classes. When using the policy gradient algorithm for parameter update, calculate the probability ratio between the current policy and the historical policy, and combine the benefit value to generate the gradient update direction. Introduce a KL divergence constraint term in this process to limit the policy update amplitude within a preset threshold range and avoid drastic fluctuations in model parameters. Through multiple rounds of iterative optimization, the model gradually adjusts the weight distribution of the feature extraction layer and the classification layer, focusing on strengthening the recognition ability in low-benefit emotion categories.

[0076] Traditional methods mostly use fixed-weight cross-entropy loss for static optimization and are difficult to dynamically adjust the optimization direction according to the data distribution. This application introduces a revenue value feedback mechanism to establish a dynamic association between data features and optimization goals, enabling the model to perform differential learning for the recognition difficulty of different emotion categories. The gradient descent method in the prior art only considers the average loss of the current batch, while the policy gradient algorithm adopted in this solution accumulates long-term revenue through trajectory sampling, enhancing the generalization ability of the model in the continuous decision-making process. This application effectively solves the problem of low optimization efficiency of traditional emotion recognition models in diverse data scenarios. By accurately positioning the weak links of the model through the dynamic revenue evaluation mechanism and combining the parameter space exploration ability of the policy gradient algorithm, the recognition accuracy of the model for mixed emotions and low-frequency emotion categories is significantly improved. This method can accurately distinguish the angry emotions of users from high-intensity background noise interference in the intelligent customer service scenario and can effectively identify the subtle emotion fluctuation characteristics related to depressive tendencies in the mental health assessment scenario, avoiding the problem of insufficient generalization ability of the model caused by traditional static optimization methods.

[0077] The objective function is as follows:

[0078]

[0079] where, π θ (o i |q) is the probability output by the fine-tuned large speech model under the current policy parameters, is the probability output by the fine-tuned large speech model under the previous policy parameters, A i is the revenue value, π ref are the model parameters of the fine-tuned large speech model, ε is the boundary parameter of the Clip function, β is the weight coefficient of the KL divergence term, and G is the number of answer samples included in each group.

[0080] The calculation method of the revenue value is as follows:

[0081]

[0082] where, r i represents the revenue of the i-th answer, 1 for correct and 0 for wrong.

[0083] In the embodiments of the present application, during operation, the policy probability distribution under the current parameters is first calculated through the policy gradient algorithm, and the old policy distribution of the previous iteration is recorded simultaneously. During each parameter update process, the objective function calculates both the benefit gain brought by policy improvement and the stability risk brought by policy distribution deviation. The Clip operation limits the probability change ratio between the new and old policies within a fixed interval. When the model attempts to perform aggressive parameter updates, this mechanism automatically suppresses gradient signals exceeding a predetermined threshold. The KL divergence constraint term monitors the model update amplitude from the perspective of probability distribution. When the response pattern difference between the new and old policies in the speech feature space exceeds the set threshold, this constraint term will generate a reverse gradient to balance the optimization direction. The dynamic calculation of the benefit value enables the model to adjust the optimization focus according to the actual emotion classification effect. For example, when identifying angry emotions in a high-noise environment, the system can automatically enhance the attention to specific spectral features.

[0084] S5: Obtain the information of the speech segment to be recognized, and perform emotion recognition on the information of the speech segment to be recognized based on the target emotion recognition model to obtain an emotion recognition result.

[0085] Specifically, during the model recommendation process or the actual application process, obtain the information of the speech segment to be recognized, perform speech noise reduction on this information to obtain the denoised speech segment information, convert the denoised speech into a speech feature vector, generate text emotion judgment information through an instruction trigger and convert it into text features, and perform feature alignment and emotion classification through the target emotion recognition model to generate an emotion recognition result.

[0086] Please refer to Figure 7 , Figure 7 which shows a specific implementation manner of step S5, described in detail as follows:

[0087] S51: Obtain the information of the speech segment to be recognized, and perform speech noise reduction on the information of the speech segment to be recognized to obtain the denoised speech segment information.

[0088] S52: Convert the denoised speech segment information into a speech feature vector.

[0089] S53: Generate text emotion judgment information through an instruction trigger, and convert the text emotion judgment information into text features.

[0090] S54: Align the speech feature vector with the text features through the target emotion recognition model, and perform emotion classification based on the aligned features to generate the emotion recognition result.

[0091] Among them, voice noise reduction refers to eliminating environmental noise interference through signal processing algorithms, which can be specifically achieved by spectral subtraction or deep learning noise reduction models, and is used to retain the effective emotional features in the voice and suppress the signal distortion caused by background noise. Voice feature vector conversion refers to converting the time-domain audio signal into a high-dimensional numerical representation, which can be specifically achieved by Mel spectrum feature extraction or pre-trained acoustic encoders, and is used to convert unstructured voice data into an input form suitable for neural network processing. Generating text emotion judgment information by instruction trigger means dynamically generating text prompts that match the current scenario according to a preset semantic template, which can be specifically achieved by rule-based keyword replacement or generative language models, and is used to provide interpretable semantic guidance information for voice emotion recognition. Feature alignment refers to establishing an implicit correlation mapping between the voice and text modalities, which can be specifically achieved by attention mechanisms or cross-modal contrast learning, and is used to eliminate the representational differences between the two modalities and establish a complementary information fusion mechanism.

[0092] Specifically, the voice to be recognized first undergoes noise reduction processing to eliminate the interference of environmental noise on emotion-related acoustic parameters such as fundamental frequency and formants. For example, in the intelligent customer service scenario, keyboard typing sounds and air conditioner noises are filtered. The noise-reduced voice extracts 80-dimensional spectral features through a Mel filter bank and is then encoded into a 128-dimensional feature vector through a convolutional neural network. At the same time, a text prompt containing emotion keywords is generated according to the preset interaction scenario, such as "Which of the following emotional states may the user be in currently", and the text is converted into a 256-dimensional text vector through a word embedding layer. The target emotion recognition model calculates the correlation weights between the voice feature vector and the text feature vector through a cross-modal attention layer. For example, when the voice features are blurred due to residual noise, the "angry" keyword in the text prompt is used to enhance the classification confidence of the corresponding emotion category. The aligned fusion features pass through a fully connected layer to output a multi-class probability distribution, and the final recognition result synthesizes the emotional expression intensity of the voice signal and the semantic orientation of the text prompt.

[0093] Traditional methods usually adopt fixed noise suppression thresholds, which lead to excessive attenuation of speech features. For example, in a factory environment, a high-pitched emotional expression may be misjudged as noise. In contrast, this application adaptively retains the information of emotion-related frequency bands through a learnable noise reduction model. For example, in a vehicle driving scenario, it retains the angry features of the user raising the volume. In the prior art, text prompts are mostly static templates. For example, "Please select the emotion category" is fixedly used as a guide, which is difficult to adapt to different scenarios such as medical consultations and children's education. This solution dynamically generates text prompts containing domain keywords through an instruction trigger mechanism. For example, in an educational scenario, it generates a guiding statement like "Which of the following learning emotions may a student show", effectively improving the generalization ability in multiple scenarios. This application can accurately identify the anxiety emotions of passengers in a continuous environmental noise scenario such as an airport waiting hall, avoiding the problem of emotion misjudgment caused by noise interference in traditional methods; in a remote medical consultation scenario, it uses dynamically generated emotion prompt words related to the condition to assist in identifying the depressive tendency of patients, solving the classification deviation problem that a single speech feature is vulnerable to network transmission distortion; in an intelligent vehicle system, it combines the driving environment to automatically generate text prompts related to driving safety, effectively distinguishing the high-intensity angry emotions of drivers caused by road rage from the excited expressions under normal circumstances.

[0094] The embodiments of this application can be applied in the medical field. Speech emotion recognition can be used for the early screening of depression and patient monitoring. For example, patients regularly record voice diaries through a mobile application. The system analyzes their speech emotions (such as speech rate, pitch, pause frequency), combines clinical data to assist doctors in diagnosis, and issues real-time warnings of emotional deterioration. In this scenario, discovering potential depressive tendencies through daily speech analysis is more objective and timely than traditional questionnaires; tracking the emotional changes of patients in real time provides data support for doctors to adjust drug doses or treatment plans; localizing the processing of voice data to avoid the leakage of sensitive information, meeting the requirements of medical ethics. The embodiments of this application can be applied in the fintech field. Speech emotion recognition can be used to improve the service quality of intelligent investment advisors. For example, when a customer interacts with an intelligent investment advisor system through voice, the system needs to analyze the customer's speech emotions (such as anxiety, hesitation, confidence) in real time, and dynamically adjust investment suggestions in combination with their investment preferences and risk tolerance. Optimizing investment strategies through emotion recognition reduces the loss risk caused by emotional decision-making of customers. Even in a noisy environment of a telephone customer service, the model can still accurately extract emotion features (such as the keyboard sound in the background noise does not affect the analysis).

[0095] In the embodiments of the present application, speech segment information and text description information corresponding to the speech segment information are obtained; the speech segment information and the text description information are subjected to feature encoding and feature alignment to generate joint features, and the pre-trained speech large model is self-supervised trained through the joint features to generate an initial speech large model; speech instruction pairs are collected, and the initial speech large model is fine-tuned based on the speech instruction pairs to obtain a fine-tuned speech large model; speech information with diverse emotion labels is obtained, and the objective function of the fine-tuned speech large model is optimized based on the speech information to generate an objective emotion recognition model; speech segment information to be recognized is obtained, and emotion recognition is performed on the speech segment information to be recognized based on the objective emotion recognition model to obtain an emotion recognition result. The embodiments of the present invention solve the problems of low recognition accuracy and insufficient feature expression ability of traditional methods in a noisy environment by training a large model with joint features that fuse speech segments and text description information, and adopt a multi-stage optimization strategy, having the advantages of improving emotion recognition accuracy, robustness, and enhancing the generalization ability of the model.

[0096] Please refer to Figure 8 , as an implementation of the above Figure 2 shown method, an embodiment of an emotion recognition device based on a speech large model is provided in the present application. This device embodiment corresponds to the Figure 2 shown method embodiment and can be specifically applied to various electronic devices.

[0097] As Figure 8 shown, the emotion recognition device based on a speech large model in this embodiment includes: an initial information acquisition module 61, a feature alignment module 62, a model fine-tuning module 63, an objective function optimization module 64, and an emotion recognition module 65, where:

[0098] The initial information acquisition module 61 is configured to obtain speech segment information and text description information corresponding to the speech segment information;

[0099] The feature alignment module 62 is configured to perform feature encoding and feature alignment on the speech segment information and the text description information to generate joint features, and perform self-supervised training on the pre-trained speech large model through the joint features to generate an initial speech large model;

[0100] The model fine-tuning module 63 is configured to collect speech instruction pairs, and perform model fine-tuning on the initial speech large model based on the speech instruction pairs to obtain a fine-tuned speech large model;

[0101] The objective function optimization module 64 is configured to obtain speech information with diverse emotion labels, and optimize the objective function of the fine-tuned speech large model based on the speech information to generate an objective emotion recognition model;

[0102] An emotion recognition module 65, configured to obtain information of a speech segment to be recognized, and perform emotion recognition on the information of the speech segment to be recognized based on the target emotion recognition model to obtain an emotion recognition result.

[0103] Further, the feature alignment module 62 includes:

[0104] A high-dimensional feature vector conversion unit, configured to convert the speech segment information into a high-dimensional feature vector through a speech encoder;

[0105] A text embedding unit, configured to convert the text description information into a text embedding vector through an embedding layer;

[0106] A feature fusion unit, configured to perform feature alignment and feature fusion on the high-dimensional feature vector and the text embedding vector to generate the joint feature;

[0107] A self-supervised training unit, configured to perform self-supervised training on the pre-trained large speech model based on the joint feature by using a cross-entropy loss function to generate the initial large speech model.

[0108] Further, the feature fusion unit includes:

[0109] A vector mapping unit, configured to map the high-dimensional feature vector and the text embedding vector to the same latent space through a linear projection layer to perform feature alignment on the high-dimensional feature vector and the text embedding vector to obtain an aligned high-dimensional feature vector and an aligned text embedding vector;

[0110] A joint feature generation unit, configured to integrate the positional encoding of the aligned high-dimensional feature vector and the positional information of the aligned text embedding vector to generate the joint feature.

[0111] Further, the model fine-tuning module 63 includes:

[0112] A full-scale fine-tuning unit, configured to collect the speech instruction pairs, and perform full-scale fine-tuning on the parameters of the initial large speech model based on the speech instruction pairs to obtain an initially fine-tuned large speech model;

[0113] A low-rank matrix insertion unit, configured to insert a low-rank matrix into the initially fine-tuned large speech model, and use an adapter to insert lightweight adaptation modules into each layer of the initially fine-tuned large speech model to obtain the fine-tuned large speech model.

[0114] Further, the objective function optimization module 64 includes:

[0115] A speech information acquisition unit, configured to acquire the speech information with diverse emotion labels;

[0116] A revenue value calculation unit, configured to perform recognition based on the voice information through the fine-tuned large voice model, generate a recognition result, and calculate a revenue value according to the recognition result and the emotion label in the voice information;

[0117] A target emotion recognition model generation unit, configured to optimize the target function based on the revenue value using a policy gradient algorithm to generate the target emotion recognition model.

[0118] Further, the target function is:

[0119]

[0120] where, π θ (o i |q) is the probability output by the fine-tuned large voice model under the current policy parameters, is the probability output by the fine-tuned large voice model under the previous policy parameters, A i is the revenue value, π ref the model parameters of the fine-tuned large voice model, ε is the boundary parameter of the Clip function, β is the weight coefficient of the KL divergence term, and G is the number of response samples included in each group.

[0121] Further, the emotion recognition module 65 includes:

[0122] A voice noise reduction unit, configured to obtain the to-be-recognized voice segment information and perform voice noise reduction on the to-be-recognized voice segment information to obtain noise-reduced voice segment information;

[0123] A voice feature vector generation unit, configured to convert the noise-reduced voice segment information into a voice feature vector;

[0124] A text feature conversion unit, configured to generate text emotion judgment information by instruction trigger and convert the text emotion judgment information into a text feature;

[0125] An emotion classification unit, configured to perform feature alignment on the voice feature vector and the text feature through the target emotion recognition model and perform emotion classification based on the aligned features to generate the emotion recognition result.

[0126] To solve the above technical problems, an embodiment of the present application further provides a computer device. For details, please refer to Figure 9 , Figure 9 which is the basic structural block diagram of the computer device in this embodiment.

[0127] The computer device 7 includes a memory 71, a processor 72, and a network interface 73 that are communicatively connected to each other through a system bus. It should be noted that, Figure 9Only a computer device 7 with three components, namely a memory 71, a processor 72, and a network interface 73, is shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Among them, those skilled in the art of the present technology can understand that the computer device here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0128] The computer device can be a desktop computer, a notebook, a palm computer, a cloud server, or other computing devices. The computer device can interact with the user through a keyboard, a mouse, a remote control, a touchpad, a voice control device, or other means.

[0129] The memory 71 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, a hard disk, a multimedia card, a card-type memory (such as an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 71 can be an internal storage unit of the computer device 7, such as the hard disk or memory of the computer device 7. In other embodiments, the memory 71 can also be an external storage device of the computer device 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the computer device 7. Of course, the memory 71 can also include both the internal storage unit and the external storage device of the computer device 7. In this embodiment, the memory 71 is generally used to store the operating system and various application software installed on the computer device 7, such as the program code of the emotion recognition method based on the speech large model. In addition, the memory 71 can also be used to temporarily store various data that have been output or will be output.

[0130] The processor 72 may be a Central Processing Unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 72 is generally used to control the overall operation of the computer device 7. In this embodiment, the processor 72 is used to run the program code stored in the memory 71 or process data, such as running the program code of the above-mentioned emotion recognition method based on the speech large model, so as to implement various embodiments of the emotion recognition method based on the speech large model.

[0131] The network interface 73 may include a wireless network interface or a wired network interface. The network interface 73 is generally used to establish a communication connection between the computer device 7 and other electronic devices.

[0132] The present application also provides another implementation manner, that is, to provide a computer-readable storage medium storing a computer program, and the computer program can be executed by at least one processor so that at least one processor executes the steps of an emotion recognition method based on a speech large model as described above.

[0133] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation manner. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of the various embodiments of the present application.

[0134] Obviously, the above-described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The drawings show the preferred embodiments of the present application, but do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure content of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or perform equivalent replacements on some of the technical features. Any equivalent structure directly or indirectly using the content of the specification and drawings of the present application in other related technical fields shall be within the scope of the patent protection of the present application by the same token.

Claims

1. A method for emotion recognition based on a large speech model, characterized in that, Including: Obtaining voice segment information and text description information corresponding to the voice segment information; Performing feature encoding and feature alignment on the voice segment information and the text description information to generate a combined feature, and performing self-supervised training on a pre-trained large voice model through the combined feature to generate an initial large voice model; Collecting voice instruction pairs, and performing model fine-tuning on the initial large voice model based on the voice instruction pairs to obtain a fine-tuned large voice model; Obtaining voice information with diverse emotion labels, and optimizing the objective function of the fine-tuned large voice model based on the voice information to generate a target emotion recognition model; Obtaining voice segment information to be recognized, and performing emotion recognition on the voice segment information to be recognized based on the target emotion recognition model to obtain an emotion recognition result.

2. The emotion recognition method based on a speech large model according to claim 1, wherein The performing feature encoding and feature alignment on the voice segment information and the text description information to generate a combined feature, and performing self-supervised training on a pre-trained large voice model through the combined feature to generate an initial large voice model includes: Converting the voice segment information into a high-dimensional feature vector through a voice encoder; Converting the text description information into a text embedding vector through an embedding layer; Performing feature alignment and feature fusion on the high-dimensional feature vector and the text embedding vector to generate the combined feature; Performing self-supervised training on the pre-trained large voice model based on the combined feature using a cross-entropy loss function to generate the initial large voice model.

3. The emotion recognition method based on a speech large model according to claim 2, wherein The performing feature alignment and feature fusion on the high-dimensional feature vector and the text embedding vector to generate the combined feature includes: Mapping the high-dimensional feature vector and the text embedding vector to the same latent space through a linear projection layer to align the features of the high-dimensional feature vector and the text embedding vector, obtaining an aligned high-dimensional feature vector and an aligned text embedding vector; Integrating the position encoding of the aligned high-dimensional feature vector with the position information of the aligned text embedding vector to generate the combined feature.

4. The emotion recognition method based on a speech large model according to claim 1, wherein The collecting voice instruction pairs, and performing model fine-tuning on the initial large voice model based on the voice instruction pairs to obtain a fine-tuned large voice model includes: Collecting the voice instruction pairs, and performing full-scale fine-tuning on the parameters of the initial large voice model based on the voice instruction pairs to obtain an initially fine-tuned large voice model; Inserting a low-rank matrix into the initially fine-tuned large voice model, and using an adaptor to insert lightweight adaptation modules into each layer of the initially fine-tuned large voice model to obtain the fine-tuned large voice model.

5. The emotion recognition method based on a speech large model according to claim 1, wherein The obtaining voice information with diverse emotion labels, and optimizing the objective function of the fine-tuned large voice model based on the voice information to generate a target emotion recognition model includes: Obtaining the voice information with diverse emotion labels; Performing recognition on the voice information through the fine-tuned large voice model to generate a recognition result, and calculating a reward value according to the recognition result and the emotion labels in the voice information; Using a policy gradient algorithm to optimize the objective function based on the reward value to generate the target emotion recognition model.

6. The emotion recognition method based on a speech large model according to claim 5, wherein The objective function is as follows: Among them, π θ (o i (q) is the probability output by the fine-tuned speech large model under the current policy parameters, is the probability output by the fine-tuned speech large model under the previous policy parameters, A i is the revenue value, π ref are the model parameters of the fine-tuned speech large model, ε is the boundary parameter of the Clip function, β is the weight coefficient of the KL divergence term, and G is the number of response samples included in each group.

7. The emotion recognition method based on a speech large model according to any one of claims 1 to 6, characterized in that, Obtaining the information of the speech segment to be recognized, and performing emotion recognition on the information of the speech segment to be recognized based on the target emotion recognition model to obtain an emotion recognition result, including: Obtaining the information of the speech segment to be recognized, and performing speech noise reduction on the information of the speech segment to be recognized to obtain the denoised speech segment information; Converting the denoised speech segment information into a speech feature vector; Generating text emotion judgment information by instruction trigger, and converting the text emotion judgment information into text features; Aligning the speech feature vector and the text features through the target emotion recognition model, and performing emotion classification based on the aligned features to generate the emotion recognition result.

8. An emotion recognition device based on a large speech model, characterized in that, Including: An initial information acquisition module, configured to acquire speech segment information and the text description information corresponding to the speech segment information; A feature alignment module, configured to perform feature encoding and feature alignment on the speech segment information and the text description information to generate combined features, and perform self-supervised training on a pre-trained large speech model through the combined features to generate an initial large speech model; A model fine-tuning module, configured to collect speech instruction pairs, and perform model fine-tuning on the initial large speech model based on the speech instruction pairs to obtain a fine-tuned large speech model; An objective function optimization module, configured to obtain speech information with diverse emotion labels, and optimize the objective function of the fine-tuned large speech model based on the speech information to generate a target emotion recognition model; An emotion recognition module, configured to obtain the information of the speech segment to be recognized, and perform emotion recognition on the information of the speech segment to be recognized based on the target emotion recognition model to obtain an emotion recognition result.

9. A computer device, characterized in that, Including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the emotion recognition method based on a large speech model according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the emotion recognition method based on a large speech model according to any one of claims 1 to 7 is implemented.

Images