A method and system for multi-emotion perception positioning of long-time speech

Abstract

The application discloses a long-time speech multi-emotion perception positioning method and system, relates to the technical field of artificial intelligence, and inputs long-time speech to be identified into a trained time sequence multi-scale space dual model, and outputs multiple emotion states in the long-time speech; the training process of the time sequence multi-scale space dual model is as follows: a training data set is constituted; the training data set is input into a time sequence multi-scale SSD architecture to obtain low-dimensional state space dual output, medium-dimensional state space dual output and high-dimensional state space dual output; the low-dimensional state space dual output, the medium-dimensional state space dual output and the high-dimensional state space dual output are subjected to feature fusion, and the complex long-time multi-emotion speech time sequence features obtained by fusion are input into an emotion analysis module to predict final multi-emotion labels and time positioning information; the multi-emotion perception positioning method and system enhance the accuracy and robustness of multi-emotion recognition.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for multi-emotion perception and localization of long-term speech. Background Technology

[0002] A multi-emotion perception localization method for long-duration speech is applicable to processing long speech segments containing multiple emotional changes, which has broad application value in fields such as affective computing, intelligent customer service, and mental health monitoring. For example, in intelligent customer service systems, a customer's emotions may fluctuate with changes in the content of the conversation. This method can locate the customer's different emotional states during the conversation in real time, thereby helping the system to make more personalized and emotional responses. Furthermore, in mental health monitoring, this method can identify and track an individual's emotional changes during long conversations or monologues, providing data support for clinical diagnosis and psychological intervention.

[0003] With the continuous enhancement of modern artificial intelligence technology and audio processing capabilities, a deep learning-based method for multi-emotion perception localization in long-term speech has encountered the following obstacles:

[0004] I. In the field of speech recognition, no comprehensive method for long-term speech emotion localization has yet been proposed. For speech emotion localization with a single emotion tag, the only existing method is Speech Emotion Segmentation (SED), proposed by Wang Yingzhi et al. at the 2023 Workshop on Automatic Speech Recognition and Understanding, titled "Speech Emotion Segmentation: When Does Emotion Appear?". While it can identify emotional changes in speech and determine their boundaries, it has limitations when processing long-term speech. The SED method mainly relies on frame-level classification and post-processing techniques to locate emotion boundaries. This method struggles to effectively capture long-term dependencies because frame-level processing has limited temporal context. Especially when processing long dialogues containing multiple emotional changes, its localization accuracy and stability are affected.

[0005] Second, in the context of long-duration speech multi-emotion label localization methods, a relatively close work is "SpeechEE: A New Benchmark for Speech Event Extraction" published by Wang Bing et al. This paper proposes the SpeechEE task, aiming to extract event information from speech and fill a gap in the field of speech event extraction. However, the main drawback of the SpeechEE task is its inability to simultaneously identify and locate emotional changes in speech. Although SpeechEE focuses on extracting event information from speech and solves the problems of event trigger word and argument recognition, it neglects the extraction of emotional information, which is an important dimension that cannot be ignored in speech understanding. Furthermore, when processing long-duration speech, the SpeechEE task lacks the ability to accurately locate dynamic emotional changes, especially in scenarios where multiple emotions alternate, its model struggles to capture subtle fluctuations in emotion over time. This makes SpeechEE exhibit significant limitations in complex speech sentiment analysis tasks that require multi-emotion label localization. Summary of the Invention

[0006] Based on the technical problems existing in the background technology, this invention proposes a multi-emotion perception and localization method and system for long-term speech, which enhances the accuracy and robustness of multi-emotion recognition.

[0007] The present invention proposes a multi-emotion perception and localization method for long-term speech, which inputs the long-term speech to be identified into a trained temporal multi-scale spatial dual model and outputs multiple emotional states in the long-term speech.

[0008] The training process of the temporal multi-scale spatial dual model is as follows:

[0009] Step 1: Obtain long-term speech of various emotions and extract them as initial long-term speech feature vectors through a general speech pre-training model to form a training dataset;

[0010] Step 2: Normalize the initial long-time speech feature vector and input it into the low-dimensional temporal SSD architecture module to obtain the low-dimensional state space dual output.

[0011] Step 3: Perform temporal downsampling and layer normalization operations on the low-dimensional state space dual output along the time dimension. Input the features that are reduced by half in the time dimension into the mid-dimensional temporal SSD architecture module to obtain the mid-dimensional state space dual output.

[0012] Step 4: Perform temporal downsampling and layer normalization operations on the mid-dimensional state space dual output along its time dimension. Input the features that are reduced by half in the time dimension into the high-dimensional temporal SSD architecture module to obtain the high-dimensional state space dual output.

[0013] Step 5: Perform feature fusion operation on the dual outputs of the low-dimensional state space, the dual outputs of the medium-dimensional state space, and the dual outputs of the high-dimensional state space. Input the fused complex long-term multi-emotional speech temporal features into the emotion parsing module to predict the final multi-emotional labels and temporal location information.

[0014] Step 6: Construct the total loss function based on the multi-emotional labels and temporal location information output in Step 5, and backpropagate to adjust the trainable parameters in the temporal multi-scale spatial dual model, thereby training the temporal multi-scale spatial dual model.

[0015] Furthermore, in steps two through five, the low-dimensional temporal SSD architecture module, the mid-dimensional temporal SSD architecture module, and the high-dimensional temporal SSD architecture module have the same structure, all including a gating mechanism, a one-dimensional convolutional layer, an activation layer, and a state-space dual model. The processing procedure of the temporal SSD architecture module for the input features is as follows:

[0016] The input features are passed through a fully connected layer and then used as input to the state-space dual model.

[0017] The input features are passed sequentially through a fully connected layer, a one-dimensional convolutional layer, and an activation layer before being used as the input to the state-space dual model.

[0018] The state-space dual output is obtained by multiplying the output of the input features through the gating mechanism with the output of the state-space dual model.

[0019] Furthermore, the formula for calculating the input features in the state-space dual model is as follows:

[0020] s t =A t s t-1 +B t x t

[0021] y t =C t s t

[0022] Among them, s t s t-1 Let A represent the hidden state vectors at times t and t-1, respectively. t B t C t The matrix parameters x and x are respectively the matrix parameters that change with time t. t y t These are the input and output features of the state-space dual model at time t, respectively.

[0023] Among them, the state-space dual model is for A. t Settings: Set all diagonal elements to the same value, and set all off-diagonal elements to zero.

[0024] Furthermore, the gating mechanism processes the input features as follows: the output of the input features after passing through the fully connected layer and the activation layer in sequence is used as the output of the gating mechanism.

[0025] Furthermore, in step five, a feature fusion operation is performed on the low-dimensional state space dual output, the mid-dimensional state space dual output, and the high-dimensional state space dual output to obtain the temporal features of complex long-term multi-emotional speech, specifically:

[0026] Linear interpolation is performed on the dual outputs of the mid-dimensional state space and the high-dimensional state space respectively to restore the same dimension as the dual output of the low-dimensional state space in the time dimension.

[0027] The low-dimensional state space dual output, the mid-dimensional state space dual output after linear interpolation, and the high-dimensional state space dual output after linear interpolation are concatenated and then subjected to linear layer dimensionality reduction to output complex long-term multi-emotional speech temporal features.

[0028] Furthermore, in step five, the emotion analysis module includes a multi-emotion classification module and a time-based positioning module. The specific processing procedure of the emotion analysis module is as follows:

[0029] The complex, long-term, multi-emotional speech temporal features are input into the multi-emotion classification module, and after average pooling and multilayer perceptron operation, multi-emotion labels are output.

[0030] Complex, long-duration, multi-emotional speech temporal features are input into the time localization module, and the time localization information is output after passing through a fully connected layer and an activation layer.

[0031] Furthermore, in step six, the total loss function is constructed as follows:

[0032] A cross-entropy loss function is constructed based on the predicted multi-emotion labels and the real emotion labels;

[0033] A mean squared error loss function is constructed based on the predicted time positioning information and the actual time positioning information.

[0034] The total loss function is constructed by summing the cross-entropy loss function and the mean squared error loss function.

[0035] A multi-emotion perception and localization system for long-term speech inputs the long-term speech to be identified into a trained temporal multi-scale spatial dual model and outputs multiple emotional states in the long-term speech.

[0036] The training process of the temporal multi-scale spatial dual model is as follows:

[0037] Step 1: Obtain long-term speech of various emotions and extract them as initial long-term speech feature vectors through a general speech pre-training model to form a training dataset;

[0038] Step 2: Normalize the initial long-time speech feature vector and input it into the low-dimensional temporal SSD architecture module to obtain the low-dimensional state space dual output.

[0039] Step 3: Perform temporal downsampling and layer normalization operations on the low-dimensional state space dual output along the time dimension. Input the features that are reduced by half in the time dimension into the mid-dimensional temporal SSD architecture module to obtain the mid-dimensional state space dual output.

[0040] Step 4: Perform temporal downsampling and layer normalization operations on the mid-dimensional state space dual output along its time dimension. Input the features that are reduced by half in the time dimension into the high-dimensional temporal SSD architecture module to obtain the high-dimensional state space dual output.

[0041] Step 5: Perform feature fusion operation on the dual outputs of the low-dimensional state space, the dual outputs of the medium-dimensional state space, and the dual outputs of the high-dimensional state space to obtain the temporal features of complex long-term multi-emotional speech. Then, input the temporal features of complex long-term multi-emotional speech into the emotion parsing module to predict the final multi-emotional labels and temporal location information.

[0042] Step 6: Construct the total loss function based on the multi-emotional labels and temporal location information output in Step 5, and backpropagate to adjust the trainable parameters in the temporal multi-scale spatial dual model, thereby training the temporal multi-scale spatial dual model.

[0043] The advantages of the long-term speech multi-emotion perception localization method and system provided by this invention are as follows: It combines a low-dimensional temporal SSD architecture module, a mid-dimensional temporal SSD architecture module, a high-dimensional temporal SSD architecture module, and an emotion parsing module to achieve multi-emotion perception localization of long-term speech sequences from a multi-scale temporal perspective. This unique architecture design allows the temporal multi-scale spatial dual model to extract rich long-term speech emotional information from multiple scales, effectively capturing both long-term and short-term temporal information, enhancing the accuracy and robustness of multi-emotion recognition, and innovatively proposing the new task of long-term speech multi-emotion perception localization. Furthermore, the introduced state-space dual model, through its selective forgetting characteristic at long-term multi-scale to retain key information, effectively solves the performance bottleneck of traditional deep learning models when processing long-term speech, improves the model's computational efficiency, and provides the possibility for realizing multi-emotion perception localization applications of long-term speech sequences. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the process of the present invention;

[0045] Figure 2 This is a structural diagram of a time-series multi-scale SSD architecture. Detailed Implementation

[0046] The technical solution of the present invention will now be described in detail through specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0047] like Figures 1 to 2 As shown, the present invention proposes a multi-emotion perception and localization method for long-term speech, which inputs the long-term speech to be identified into a trained temporal multi-scale spatial dual model and outputs multiple emotional states in the long-term speech.

[0048] The training process of the temporal multi-scale spatial dual model is as follows:

[0049] Step 1: Obtain long-term speech of various emotions and extract them as initial long-term speech feature vectors through a general speech pre-training model to form a training dataset;

[0050] We acquire long-term speech containing multiple emotions as input, and use a general language pre-trained model (such as the Hubert speech pre-trained model) to convert it into an initial long-term speech feature vector, denoted as... in Let T represent the length of the long-term speech sequence and D represent the feature dimension. These feature vectors together constitute the training dataset required to train the temporal multi-scale spatial dual model (temporal multi-scale SSD architecture model). Through self-supervised learning and massive training data, the Hubert speech pre-training model not only deeply explores the complex features in speech but also significantly improves the accuracy of downstream tasks such as speech emotion recognition and speech synthesis.

[0051] In the i-th training epoch, the initial long-term speech feature vector is input into the temporal multi-scale spatial dual model. During the training process based on the temporal multi-scale spatial dual model, training datasets with the same batch will be trained in different training epochs. The training epoch can be initialized to i=1, and the training epoch can be iterated in i=i+1. Here, let the parameter based on the temporal multi-scale SSD architecture in the i-th training epoch be β. i .

[0052] like Figure 2As shown, in this embodiment, the temporal multi-scale SSD architecture includes a low-dimensional temporal SSD architecture module, a mid-dimensional temporal SSD architecture module, and a high-dimensional temporal SSD architecture module. These three temporal SSD architecture modules have the same structure, all including a gating mechanism, a one-dimensional convolutional layer, an activation layer, and a state-space dual model. The temporal SSD architecture module processes the input features as follows: the input features pass through a fully connected layer and are then used as the input to the state-space dual model; the input features then pass through a fully connected layer, a one-dimensional convolutional layer, and an activation layer sequentially and are then used as the input to the state-space dual model; the output of the input features after passing through the gating mechanism is multiplied by the output of the state-space dual model to obtain the state-space dual output. See steps two to four for details.

[0053] Step 2: Normalize the initial long-time speech feature vector and input it into the low-dimensional temporal SSD architecture module to obtain the low-dimensional state space dual output.

[0054] (a1) During the i-th training iteration, the low-dimensional temporal SSD architecture module first uses equation (1) to process the initial long-term speech feature vector x. i Layer normalization is performed to obtain the input features of the low-dimensional temporal SSD architecture module.

[0055]

[0056] In equation (1), norm(·) represents the layer normalization operation, which can make the temporal multi-scale spatial dual model more adaptable to different input data, thereby improving the generalization ability of the temporal multi-scale spatial dual model.

[0057] (a2) During the i-th training iteration, the low-dimensional temporal SSD architecture module calculates the input features using equation (2). The low-dimensional state-space dual output obtained after passing through the low-dimensional temporal SSD architecture module

[0058] In equation (2), W k (·) represents a linear fully connected layer, W c (·) denotes a one-dimensional convolutional layer, Silu(·) denotes the SiLU activation function, SSD(·) denotes the state-space dual model, ⊙ denotes the Hadamard product operation of matrices, and Gated(·) denotes the gating mechanism in the state-space dual architecture (SSD architecture), and has the following:

[0059]

[0060] W in equations (2) and (3) lThe linear fully connected layer can be represented by two different settings, or it can be the same linear fully connected layer. For the sake of convenience, this embodiment uses the same parameter.

[0061] The state-space dual model SSD(·) in equation (2) is described as follows: SSD, as an effective enhancement of the basic state-space model (SSM), introduces a structured dual mask matrix to describe the evolution of the state space. Furthermore, SSD is more efficient and has fewer model parameters than SSM. Both SSD and SSM are derived from equation (4):

[0062] s t =A t s t-1 +B t x t ;y t =C t s t (4);

[0063] In equation (4), The matrix parameters vary with time t. and Let s represent the input and output features of the state-space dual model at time t, respectively. t This represents the hidden state vector. The State-Space Dual Model (SSD) simplifies matrix A by setting all diagonal elements to the same value and all off-diagonal elements to zero. t This significantly reduces the complexity of the model, optimizes computational efficiency, and retains the ability to effectively capture the temporal and frequency characteristics of the input signal.

[0064] Step 3: Perform temporal downsampling and layer normalization operations on the low-dimensional state space dual output along the time dimension. Input the features that are reduced by half in the time dimension into the mid-dimensional temporal SSD architecture module to obtain the mid-dimensional state space dual output.

[0065] (b1) During the i-th training iteration, the mid-dimensional temporal SSD architecture module utilizes equation (5) to perform dual output on the low-dimensional state space. Temporal downsampling and layer normalization operations are performed to obtain the input of the mid-dimensional temporal SSD architecture module, which is reduced by half in the time dimension.

[0066]

[0067] In Equation (5), Downsampled(·) represents the temporal downsampling operation. The temporal downsampling operation retains key information by aggregating temporal features, while reducing the time dimension at a low-dimensional scale and lowering the feature resolution, thereby improving model efficiency and capturing global temporal patterns, providing a compact and representative feature representation for emotion recognition.

[0068] (b2) The mid-dimensional time-series SSD architecture module calculates using equation (6) during the i-th training iteration. The mid-dimensional state space dual output obtained after the mid-dimensional temporal SSD architecture module

[0069]

[0070] In equation (6), the gating mechanism is shown in equation (7):

[0071]

[0072] Step 4: Perform temporal downsampling and layer normalization operations on the mid-dimensional state space dual output along its time dimension. Input the features that are reduced by half in the time dimension into the high-dimensional temporal SSD architecture module to obtain the high-dimensional state space dual output.

[0073] (c1) During the i-th training iteration, the high-dimensional temporal SSD architecture module utilizes equation (8) to output the dual state space of the mid-dimensional state space. Temporal downsampling and layer normalization are performed along its time dimension to obtain the input of a high-dimensional temporal SSD architecture module that is halved in time dimension.

[0074]

[0075] (c2) The high-dimensional temporal SSD architecture module calculates using equation (9) during the i-th training iteration. The high-dimensional state-space dual output obtained after the high-dimensional temporal SSD architecture module

[0076]

[0077] In equation (9), the gating mechanism Gated(·) is shown in equation (10):

[0078]

[0079] Among them, W in equations (2), (6), and (9) l W c Gated, norm, W lSSD and Silu correspond to different layers in different time-series SSD architecture modules (low-dimensional, mid-dimensional, and high-dimensional). For ease of description, this embodiment uses the same parameter description.

[0080] Step 5: Perform feature fusion operation on the dual outputs of the low-dimensional state space, the dual outputs of the medium-dimensional state space, and the dual outputs of the high-dimensional state space. Input the fused complex long-term multi-emotional speech temporal features into the emotion parsing module to predict the final multi-emotional labels and temporal location information.

[0081] The generation process of complex, long-duration, multi-emotional speech temporal features is shown in (d1) to (d2):

[0082] (d1) Using equation (11), the dual outputs of the medium-dimensional state space and the dual outputs of the high-dimensional state space are linearly interpolated to restore them to the same dimension size as the dual outputs of the low-dimensional state space in the time dimension;

[0083]

[0084] In equation (11), Linearinterpolation(·) represents the linear interpolation operation. and This represents the restoration of the dual output of the mid-dimensional state space and the dual output of the high-dimensional state space to the same dimension as the dual output of the low-dimensional state space in the time dimension.

[0085] (d2) Calculate the mid-dimensional state space dual output after linear interpolation using equation (12). High-dimensional state space dual output Dual output with low-dimensional state space Complex long-term, multi-emotional speech temporal features M are obtained through concatenation and linear layer dimensionality reduction operations. i ;

[0086]

[0087] In equation (12), Concat(·) represents the splicing operation of the dual outputs of the state space at multiple scales along the feature dimension.

[0088] During the i-th training iteration, the emotion parsing module uses equation (13) to process the complex long-term multi-emotion speech features M obtained through the temporal multi-scale SSD architecture. i The data is fed into the emotion parsing module to obtain the final multi-emotion tag Emo. i and its corresponding time location information Tms i :

[0089] Emo i ,Tms i=Emotionparser(M i (13);

[0090] In equation (13), Emotionparser(·) represents the emotion parsing module, which is divided into two parts: the multi-emotion classification module Mutiemoclassifier(·) and the time location module Temlocation(·).

[0091] Emo i =Mutiemoclassifier(M i =MLP(Avgpool(M i ));

[0092] TMS i =Temlocation(M i =Softmax(W) c (M i )) (14);

[0093] In equation (14), MLP(·) represents a multilayer perceptron, Avgpool(·) represents the average pooling operation, Softmax is the activation function, i.e., the activation layer, W c It is a fully connected layer.

[0094] Step 6: Construct the total loss function based on the multi-emotional labels and temporal location information output in Step 5, and backpropagate to adjust the trainable parameters in the temporal multi-scale spatial dual model, thereby training the temporal multi-scale spatial dual model.

[0095] Using Equation (15), the multi-emotion labels Emo obtained from the temporal multi-scale spatial dual model during the i-th training time are... i Time-based location information (TMS) i This, combined with corresponding real-world multi-emotion label results and temporal location information, utilizes cross-entropy error loss and mean squared error loss functions to constrain the effectiveness of the temporal multi-scale spatial dual model in locating multi-emotion perception, guiding the optimization of the temporal multi-scale spatial dual model to improve prediction accuracy.

[0096]

[0097] Emo in equation (15) i This represents the multi-emotion label during the i-th training iteration. Tms represents true emotional tags. i This represents the temporal positioning information during the i-th training iteration. This indicates the actual time and location information. Represents the cross-entropy loss function. Let represent the mean squared error loss function. The mathematical expression for a set of N training samples can be expressed as:

[0098]

[0099] Finally, determine the total loss function Loss(ξ) for the i-th training iteration. t If the model converges, i.e. the loss value tends to remain constant, then the parameters at the i-th training time are the optimal parameters. These parameters are then used as the final temporal multi-scale spatial dual model for long-term speech multi-emotion perception and localization. Otherwise, the i-th gradient backpropagation is performed, the parameters are updated, and the model parameters at the i+1-th training time are obtained. The i+1 is assigned to i, and the process returns to step two and executes sequentially until the temporal multi-scale spatial dual model converges.

[0100] Through steps one through six, this embodiment combines a low-dimensional temporal SSD architecture module, a mid-dimensional temporal SSD architecture module, a high-dimensional temporal SSD architecture module, and an emotion parsing module to achieve multi-emotion perception localization of long-term speech sequences from a multi-scale temporal perspective. This unique architectural design enables the temporal multi-scale spatial dual model to extract rich long-term speech emotional information from multiple scales, effectively capturing both long-term and short-term temporal information, enhancing the accuracy and robustness of multi-emotion recognition, and innovatively proposing the new task of long-term speech multi-emotion perception localization. Furthermore, the introduced state-space dual model, through its selective forgetting at multiple long-term scales to retain key information, effectively solves the performance bottleneck of traditional deep learning models when processing long-term speech, improves the model's computational efficiency, and provides the possibility for realizing multi-emotion perception localization applications of long-term speech sequences.

[0101] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multi-emotion perception localization method for long-term speech, characterized in that, The long-term speech to be recognized is input into a trained temporal multi-scale spatial dual model, which outputs multiple emotional states in the long-term speech. The training process of the temporal multi-scale spatial dual model is as follows: Step 1: Obtain long-term speech of various emotions and extract them as initial long-term speech feature vectors through a general speech pre-training model to form a training dataset; Step 2: Normalize the initial long-time speech feature vector and input it into the low-dimensional temporal SSD architecture module to obtain the low-dimensional state space dual output. Step 3: Perform temporal downsampling and layer normalization operations on the low-dimensional state space dual output along the time dimension. Input the features that are reduced by half in the time dimension into the mid-dimensional temporal SSD architecture module to obtain the mid-dimensional state space dual output. Step 4: Perform temporal downsampling and layer normalization operations on the mid-dimensional state space dual output along its time dimension. Input the features that are reduced by half in the time dimension into the high-dimensional temporal SSD architecture module to obtain the high-dimensional state space dual output. Step 5: Perform feature fusion operation on the dual outputs of the low-dimensional state space, the dual outputs of the medium-dimensional state space, and the dual outputs of the high-dimensional state space. Input the fused complex long-term multi-emotional speech temporal features into the emotion parsing module to predict the final multi-emotional labels and temporal location information. Step 6: Construct the total loss function based on the multi-emotional labels and temporal location information output in Step 5, and backpropagate to adjust the trainable parameters in the temporal multi-scale spatial dual model, thereby training the temporal multi-scale spatial dual model.

2. The multi-emotion perception and localization method for long-term speech according to claim 1, characterized in that, In steps two through five, the low-dimensional temporal SSD architecture module, the mid-dimensional temporal SSD architecture module, and the high-dimensional temporal SSD architecture module have the same structure, all including a gating mechanism, a one-dimensional convolutional layer, an activation layer, and a state-space dual model. The processing procedure of the temporal SSD architecture module for the input features is as follows: The input features are passed through a fully connected layer and then used as input to the state-space dual model. The input features are passed sequentially through a fully connected layer, a one-dimensional convolutional layer, and an activation layer before being used as the input to the state-space dual model. The state-space dual output is obtained by multiplying the output of the input features through the gating mechanism with the output of the state-space dual model.

3. The multi-emotion perception and localization method for long-term speech according to claim 2, characterized in that, The formula for calculating the input features in the state-space dual model is as follows: s t =A t s t-1 +B t x t y t =C t s t Among them, s t ,s t-1 Let A represent the hidden state vectors at times t and t-1, respectively. t B t C t The matrix parameters x and x are respectively the matrix parameters that change with time t. t ,y t These are the input and output features of the state-space dual model at time t, respectively. Among them, the state-space dual model is for A. t Settings: Set all diagonal elements to the same value, and set all off-diagonal elements to zero.

4. The multi-emotion perception and localization method for long-term speech according to claim 2, characterized in that, The gating mechanism processes the input features as follows: the output of the input features after passing through the fully connected layer and the activation layer in sequence is used as the output of the gating mechanism.

5. The multi-emotion perception and localization method for long-term speech according to claim 1, characterized in that, In step five, a feature fusion operation is performed on the low-dimensional state space dual output, the mid-dimensional state space dual output, and the high-dimensional state space dual output to obtain the temporal features of complex long-term multi-emotional speech, specifically: Linear interpolation is performed on the dual outputs of the mid-dimensional state space and the high-dimensional state space respectively to restore the same dimension as the dual output of the low-dimensional state space in the time dimension. The low-dimensional state space dual output, the mid-dimensional state space dual output after linear interpolation, and the high-dimensional state space dual output after linear interpolation are concatenated and then subjected to linear layer dimensionality reduction to output complex long-term multi-emotional speech temporal features.

6. The multi-emotion perception and localization method for long-term speech according to claim 1, characterized in that, In step five, the emotion analysis module includes a multi-emotion classification module and a time-based positioning module. The specific processing procedure of the emotion analysis module is as follows: The complex, long-term, multi-emotional speech temporal features are input into the multi-emotion classification module, and after average pooling and multilayer perceptron operation, multi-emotion labels are output. Complex, long-duration, multi-emotional speech temporal features are input into the time localization module, and the time localization information is output after passing through a fully connected layer and an activation layer.

7. The multi-emotion perception and localization method for long-term speech according to claim 6, characterized in that, In step six, the total loss function is constructed as follows: A cross-entropy loss function is constructed based on the predicted multi-emotion labels and the real emotion labels; A mean squared error loss function is constructed based on the predicted time positioning information and the actual time positioning information. The total loss function is constructed by summing the cross-entropy loss function and the mean squared error loss function.

8. A multi-emotion perception and localization system for long-duration speech, characterized in that, The long-term speech to be recognized is input into a trained temporal multi-scale spatial dual model, which outputs multiple emotional states in the long-term speech. The training process of the temporal multi-scale spatial dual model is as follows: Step 1: Obtain long-term speech of various emotions and extract them as initial long-term speech feature vectors through a general speech pre-training model to form a training dataset; Step 2: Normalize the initial long-time speech feature vector and input it into the low-dimensional temporal SSD architecture module to obtain the low-dimensional state space dual output. Step 3: Perform temporal downsampling and layer normalization operations on the low-dimensional state space dual output along the time dimension. Input the features that are reduced by half in the time dimension into the mid-dimensional temporal SSD architecture module to obtain the mid-dimensional state space dual output. Step 4: Perform temporal downsampling and layer normalization operations on the mid-dimensional state space dual output along its time dimension. Input the features that are reduced by half in the time dimension into the high-dimensional temporal SSD architecture module to obtain the high-dimensional state space dual output. Step 5: Perform feature fusion operation on the dual outputs of the low-dimensional state space, the dual outputs of the medium-dimensional state space, and the dual outputs of the high-dimensional state space to obtain the temporal features of complex long-term multi-emotional speech. Then, input the temporal features of complex long-term multi-emotional speech into the emotion parsing module to predict the final multi-emotional labels and temporal location information. Step 6: Construct the total loss function based on the multi-emotional labels and temporal location information output in Step 5, and backpropagate to adjust the trainable parameters in the temporal multi-scale spatial dual model, thereby training the temporal multi-scale spatial dual model.

Images