Multi-emotion perception positioning method and system based on long-time voice

Abstract

The invention discloses a multi-emotion perception positioning method and system for long-time voice, and relates to the technical field of artificial intelligence, the long-time voice to be recognized is input into a trained time sequence multi-scale space dual model, and a plurality of emotion states in the long-time voice are output; the training process of the time sequence multi-scale space dual model is as follows: forming a training data set; inputting the training data set into the time sequence multi-scale SSD architecture to obtain low-dimensional state space dual output, medium-dimensional state space dual output and high-dimensional state space dual output; feature fusion is carried out on the low-dimensional state space dual output, the medium-dimensional state space dual output and the high-dimensional state space dual output, and complex long-time multi-emotion voice time sequence features obtained through fusion are input into an emotion analysis module to predict final multi-emotion labels and time positioning information; according to the multi-emotion perception positioning method and system, the accuracy and robustness of multi-emotion recognition are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multi-emotion perception and positioning method and system for long-duration speech. Background Art

[0002] A multi-emotion perception and localization method for long-duration speech is suitable for processing long-duration speech segments containing multiple emotional changes. It has wide application value in the fields of affective computing, intelligent customer service, and mental health monitoring. For example, in an intelligent customer service system, the customer's emotions may fluctuate as the content of the conversation changes. Using this method, the customer's different emotional states during the conversation can be located in real time, thereby helping the system to make a more personalized and emotional response. In addition, in mental health monitoring, this method can identify and track the emotional changes of individuals in long conversations or monologues, providing data support for clinical diagnosis and psychological intervention.

[0003] With the continuous enhancement of modern artificial intelligence technology and audio processing capabilities, a multi-emotion perception and localization method for long-term speech based on deep learning has encountered the following obstacles:

[0004] 1. In the field of speech, no perfect method for localizing multiple emotions in long-term speech has been proposed. For speech emotion localization with a single emotion label, the only method left is the speech emotion localization (SED) method proposed by Wang Yingzhi et al. in the 2023 Automatic Speech Recognition and Understanding Symposium in "Speech Emotion Segmentation: When Does Emotion Appear?" Although it can identify emotional changes in speech and determine its boundaries, it has limitations when processing long-term speech. The SED method mainly relies on frame-level classification and post-processing technology to locate emotional boundaries. This method is difficult to effectively capture long-term dependencies because the temporal context of frame-level processing is limited, especially when processing long conversations with multiple emotional changes, its positioning accuracy and stability will be affected.

[0005] Second, in the context of the method of locating multiple emotion labels in long-term speech, there is another relatively close work, "SpeechEE: A New Benchmark for Speech Event Extraction" published by Wang Bing et al., which proposed the SpeechEE task, which aims to extract event information from speech and fill the gap in the field of speech event extraction. However, the main disadvantage of the SpeechEE task is that it cannot simultaneously identify and locate emotional changes in speech. Although SpeechEE focuses on extracting event information from speech and solves the problem of event trigger words and argument recognition, it ignores the extraction of emotional information, which is an important dimension that cannot be ignored in speech understanding. In addition, when processing long-term speech, the SpeechEE task lacks the ability to accurately locate the dynamic changes of emotions, especially in scenarios where multiple emotions appear alternately, and its model has difficulty capturing the subtle fluctuations of emotions over time. This makes SpeechEE show obvious limitations in complex speech sentiment analysis tasks that require the location of multiple emotion labels. Summary of the invention

[0006] Based on the technical problems existing in the background technology, the present invention proposes a multi-emotion perception and positioning method and system for long-term speech, which enhances the accuracy and robustness of multi-emotion recognition.

[0007] The present invention proposes a multi-emotion perception and localization method for long-duration speech, which inputs the long-duration speech to be recognized into a trained time-series multi-scale space dual model, and outputs multiple emotional states in the long-duration speech;

[0008] The training process of the temporal multiscale spatial dual model is as follows:

[0009] Step 1: Obtain long-term speech of multiple emotions, and extract them into initial long-term speech feature vectors respectively through a general speech pre-training model to form a training data set;

[0010] Step 2: normalize the initial long-term speech feature vector and input it into the low-dimensional time series SSD architecture module to obtain a low-dimensional state space dual output;

[0011] Step 3: Perform temporal downsampling and layer normalization operations on the low-dimensional state space dual output along the time dimension, and input the features reduced by half in the time dimension into the medium-dimensional temporal SSD architecture module to obtain the medium-dimensional state space dual output;

[0012] Step 4: Perform time-series downsampling and layer normalization operations on the medium-dimensional state space dual output along its time dimension, and input the features reduced by half in the time dimension into the high-dimensional time series SSD architecture module to obtain the high-dimensional state space dual output;

[0013] Step 5: perform feature fusion operation on the low-dimensional state space dual output, the medium-dimensional state space dual output and the high-dimensional state space dual output, and input the fused complex long-term multi-emotion speech time series features into the emotion analysis module to predict the final multi-emotion labels and time positioning information;

[0014] Step 6: Construct a total loss function based on the multi-emotion labels and time positioning information output in step 5, and back-propagate to adjust the trainable parameters in the time series multi-scale spatial dual model, thereby training the time series multi-scale spatial dual model.

[0015] Furthermore, in steps 2 to 5, the structures of the low-dimensional time series SSD architecture module, the medium-dimensional time series SSD architecture module and the high-dimensional time series SSD architecture module are consistent, all including a gating mechanism, a one-dimensional convolutional layer, an activation layer and a state space dual model. The time series SSD architecture module processes the input features as follows:

[0016] The input features pass through the fully connected layer as the input of the state-space dual model;

[0017] The input features are sequentially passed through the fully connected layer, the one-dimensional convolutional layer, and the activation layer as the input of the state-space dual model;

[0018] The state-space dual output is obtained by multiplying the output of the input feature through the gating mechanism with the output of the state-space dual model.

[0019] Furthermore, the calculation formula of the state space dual model for the input features is as follows:

[0020] s t =A t s t-1 +B t x t

[0021] y t =C t s t

[0022] Among them, s t ,s t-1 Represent the hidden state vector at time t and t-1 respectively, A t , B t , C t are the matrix parameters that change with time t, x t ,y t are the input features and output features of the state space dual model at time t respectively;

[0023] Among them, the state space dual model is A t Set: Set all diagonal elements to the same value and all off-diagonal elements to zero.

[0024] Furthermore, the processing process of the gating mechanism on the input features is: the output of the input features after passing through the fully connected layer and the activation layer in sequence is used as the output of the gating mechanism.

[0025] Furthermore, in step five, the low-dimensional state space dual output, the medium-dimensional state space dual output and the high-dimensional state space dual output are subjected to feature fusion operation to obtain complex long-term multi-emotion speech time series features, specifically:

[0026] Perform linear interpolation on the dual output of the medium-dimensional state space and the dual output of the high-dimensional state space respectively, so as to restore them to the same dimension as the dual output of the low-dimensional state space in the time dimension;

[0027] The low-dimensional state space dual output, the medium-dimensional state space dual output after linear interpolation, and the high-dimensional state space dual output after linear interpolation are concatenated and subjected to a linear layer dimensionality reduction operation to output complex long-term multi-emotional speech time series features.

[0028] Further, in step 5, the emotion analysis module includes a multi-emotion classification module and a time positioning module. The processing process of the emotion analysis module is specifically as follows:

[0029] The complex long-term multi-emotion speech time series features are input into the multi-emotion classification module, and then the multi-emotion labels are output after average pooling and multi-layer perceptron operation.

[0030] The complex, long-term, and multi-emotional speech temporal features are input into the time localization module, and are operated through the fully connected layer and the activation layer to output the time localization information.

[0031] Furthermore, in step six, the total loss function is constructed as:

[0032] Construct a cross entropy loss function based on the predicted output multiple emotion labels and the true emotion labels;

[0033] Construct a mean square error loss function based on the predicted output time location information and the actual time location information;

[0034] The total loss function is constructed based on the sum of the cross entropy loss function and the mean square error loss function.

[0035] A multi-emotion perception and positioning system for long-duration speech, which inputs the long-duration speech to be recognized into a trained time-series multi-scale space dual model, and outputs multiple emotional states in the long-duration speech;

[0036] The training process of the temporal multiscale spatial dual model is as follows:

[0037] Step 1: Obtain long-term speech of multiple emotions, and extract them into initial long-term speech feature vectors respectively through a general speech pre-training model to form a training data set;

[0038] Step 2: normalize the initial long-term speech feature vector and input it into the low-dimensional time series SSD architecture module to obtain a low-dimensional state space dual output;

[0039] Step 3: Perform temporal downsampling and layer normalization operations on the low-dimensional state space dual output along the time dimension, and input the features reduced by half in the time dimension into the medium-dimensional temporal SSD architecture module to obtain the medium-dimensional state space dual output;

[0040] Step 4: Perform time-series downsampling and layer normalization operations on the medium-dimensional state space dual output along its time dimension, and input the features reduced by half in the time dimension into the high-dimensional time series SSD architecture module to obtain the high-dimensional state space dual output;

[0041] Step 5: perform feature fusion operation on the low-dimensional state space dual output, the medium-dimensional state space dual output and the high-dimensional state space dual output to obtain complex long-term multi-emotion speech time series features, and input the complex long-term multi-emotion speech time series features into the emotion analysis module to predict the final multi-emotion labels and time positioning information;

[0042] Step 6: Construct a total loss function based on the multi-emotion labels and time positioning information output in step 5, and back-propagate to adjust the trainable parameters in the time series multi-scale spatial dual model, thereby training the time series multi-scale spatial dual model.

[0043] The advantages of the multi-emotion perception and positioning method and system for long-term speech provided by the present invention are that the low-dimensional time series SSD architecture module, the medium-dimensional time series SSD architecture module, the high-dimensional time series SSD architecture module and the emotion analysis module are combined to realize the multi-emotion perception and positioning of long-term speech sequences from the perspective of time series multi-scale. This unique architectural design enables the time series multi-scale spatial dual model to extract rich long-term speech emotional information from multiple scales, effectively capture long-term and short-term time information, enhance the accuracy and robustness of multi-emotion recognition, and innovatively propose the new task of multi-emotion perception and positioning of long-term speech. In addition, the introduced state-space dual model effectively solves the performance bottleneck of traditional deep learning models in processing long-term speech through its characteristic of selectively forgetting in long-term multi-scale to retain key information, improves the computational efficiency of the model, and provides the possibility for realizing the application of multi-emotion perception and positioning of long-term speech sequences. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic diagram of the process of the present invention;

[0045] Figure 2 The structural diagram of the temporal multi-scale SSD architecture. DETAILED DESCRIPTION

[0046] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.

[0047] like Figures 1 to 2 As shown, the present invention proposes a multi-emotion perception and positioning method for long-term speech, which inputs the long-term speech to be recognized into a trained temporal multi-scale spatial dual model and outputs multiple emotional states in the long-term speech.

[0048] The training process of the temporal multiscale spatial dual model is as follows:

[0049] Step 1: Obtain long-term speech of multiple emotions, and extract them into initial long-term speech feature vectors respectively through a general speech pre-training model to form a training data set;

[0050] Take the long-term speech containing multiple emotions as input, and use the general language pre-training model (such as Hubert speech pre-training model) to convert it into the initial long-term speech feature vector, denoted as in represents the real number domain, T represents the length of the long-term speech sequence, and D represents the size of the feature dimension. These feature vectors are combined to form the training data set required for training the time series multi-scale spatial dual model (time series multi-scale SSD architecture model). The Hubert speech pre-training model not only deeply explores the complex features in speech through self-supervised learning and massive training data, but also significantly improves the accuracy of downstream tasks such as speech emotion recognition and speech synthesis.

[0051] In the i-th training round, the initial long-term speech feature vector is input into the temporal multi-scale spatial dual model. In the training process based on the temporal multi-scale spatial dual model, the same batch of training data sets will be trained in different training rounds. The training round i=1 can be initialized and the training round iteration is performed at i=i+1. Here, the parameter based on the temporal multi-scale SSD architecture in the i-th training is β i .

[0052] like Figure 2As shown, in this embodiment, the temporal multi-scale SSD architecture includes a low-dimensional temporal SSD architecture module, a medium-dimensional temporal SSD architecture module, and a high-dimensional temporal SSD architecture module. The structures of these three temporal SSD architecture modules are consistent, and all include a gating mechanism, a one-dimensional convolutional layer, an activation layer, and a state-space dual model. The temporal SSD architecture module processes the input features as follows: the input features pass through the fully connected layer as the input of the state-space dual model; the input features pass through the fully connected layer, the one-dimensional convolutional layer, and the activation layer in turn as the input of the state-space dual model; the output of the input features after the gating mechanism is multiplied with the output of the state-space dual model to obtain the state-space dual output. See steps 2 to 4 for details.

[0053] Step 2: normalize the initial long-term speech feature vector and input it into the low-dimensional time series SSD architecture module to obtain a low-dimensional state space dual output;

[0054] (a1) During the i-th training, the low-dimensional temporal SSD architecture module first uses formula (1) to calculate the initial long-term speech feature vector x i Perform layer normalization to obtain the input features of the low-dimensional temporal SSD architecture module

[0055]

[0056] In formula (1), norm(·) represents the layer normalization operation, which can make the time series multi-scale spatial dual model more adaptable to different input data, thereby improving the generalization ability of the time series multi-scale spatial dual model.

[0057] (a2) The low-dimensional temporal SSD architecture module uses formula (2) to calculate the input features during the i-th training The low-dimensional state space dual output obtained after the low-dimensional time series SSD architecture module

[0058] In formula (2), W k (·) represents a linear fully connected layer, W c (·) represents a one-dimensional convolutional layer, Silu(·) represents the SiLU activation function, SSD(·) represents the state-space dual model, ⊙ represents the Hadamard product operation of the matrix, Gated(·) represents the gating mechanism in the state-space dual architecture (SSD architecture), and:

[0059]

[0060] W in formula (2) and formula (3) lIt can be two linear fully connected layers with different settings, or it can be the same linear fully connected layer. For the convenience of description, this embodiment uses the same parameter description.

[0061] The state space dual model SSD(·) in equation (2) is introduced as follows: SSD is an effective enhancement of the basic state space model (SSM). It introduces a structured dual mask matrix to describe the evolution of the state space. Compared with SSM, SSD is more efficient and has fewer model parameters. Both SSD and SSM are derived from equation (4):

[0062] s t =A t s t-1 +B t x t ;y t =C t s t (4);

[0063] In formula (4), is the matrix parameter that changes with time t, and They represent the input and output features of the state-space dual model at time t, respectively, and s t represents the hidden state vector. The state-space dual model SSD simplifies the matrix A by setting all diagonal elements to the same value and all off-diagonal elements to zero. t , which significantly reduces the complexity of the model and optimizes the computational efficiency while retaining the ability to effectively capture the time and frequency characteristics of the input signal.

[0064] Step 3: Perform temporal downsampling and layer normalization operations on the low-dimensional state space dual output along the time dimension, and input the features reduced by half in the time dimension into the medium-dimensional temporal SSD architecture module to obtain the medium-dimensional state space dual output;

[0065] (b1) The medium-dimensional temporal SSD architecture module uses formula (5) to output the low-dimensional state space dual in the i-th training. Perform temporal downsampling and layer normalization operations to obtain the input of the medium-dimensional temporal SSD architecture module that is reduced by half in the time dimension

[0066]

[0067] In formula (5), Downsampled(·) represents the temporal downsampling operation. The temporal downsampling operation retains key information by aggregating temporal features, while reducing the time dimension at a low dimensional scale and reducing the feature resolution, thereby improving the model efficiency and capturing the global temporal pattern, providing a compact and representative feature expression for emotion recognition.

[0068] (b2) The medium-dimensional temporal SSD architecture module uses formula (6) to calculate during the i-th training The medium-dimensional state space dual output obtained after passing through the medium-dimensional time-series SSD architecture module

[0069]

[0070] In formula (6), the gating mechanism is shown in formula (7):

[0071]

[0072] Step 4: Perform time-series downsampling and layer normalization operations on the medium-dimensional state space dual output along its time dimension, and input the features reduced by half in the time dimension into the high-dimensional time series SSD architecture module to obtain the high-dimensional state space dual output;

[0073] (c1) The high-dimensional time series SSD architecture module uses formula (8) to output the dual of the medium-dimensional state space during the i-th training. Perform temporal downsampling and layer normalization operations along its time dimension to obtain the input of the high-dimensional temporal SSD architecture module that is reduced by half in the time dimension.

[0074]

[0075] (c2) The high-dimensional time series SSD architecture module uses formula (9) to calculate during the i-th training The high-dimensional state space dual output obtained after passing through the high-dimensional time series SSD architecture module

[0076]

[0077] In formula (9), the gating mechanism Gated(·) is shown in formula (10):

[0078]

[0079] For W in formula (2), formula (6), and formula (9), l , W c 、Gated、norm、W l, SSD, and Silu correspond to different layers in different sequential SSD architecture modules (low-dimensional, medium-dimensional, and high-dimensional). For the convenience of description, this embodiment uses the same parameter description.

[0080] Step 5: perform feature fusion operation on the low-dimensional state space dual output, the medium-dimensional state space dual output and the high-dimensional state space dual output, and input the fused complex long-term multi-emotion speech time series features into the emotion analysis module to predict the final multi-emotion labels and time positioning information;

[0081] The generation process of complex long-term multi-emotion speech temporal features is as follows (d1) to (d2):

[0082] (d1) Using equation (11), linearly interpolate the medium-dimensional state space dual output and the high-dimensional state space dual output to restore them to the same dimension as the low-dimensional state space dual output in the time dimension;

[0083]

[0084] In formula (11), Linearinterpolation(·) represents the linear interpolation operation, and Represents the medium-dimensional state space dual output and the high-dimensional state space dual output restored to the same dimension as the low-dimensional state space dual output in the time dimension.

[0085] (d2) Use equation (12) to calculate the dual output of the medium-dimensional state space after linear interpolation High-dimensional state space dual output Dual output with low-dimensional state space After concatenation and linear layer dimension reduction, the complex long-term multi-emotion speech temporal feature M is obtained. i ;

[0086]

[0087] In formula (12), Concat(·) represents the concatenation operation of the state space dual outputs at multiple scales along the feature dimension.

[0088] During the i-th training, the emotion analysis module uses formula (13) to convert the complex long-term multi-emotion speech features M obtained through the temporal multi-scale SSD architecture into i Passed into the emotion analysis module to obtain the final multi-emotion label Emo i And its corresponding time positioning information Tms i :

[0089] Emo i ,Tms i=Emotionparser(M i ) (13);

[0090] In formula (13), Emotionparser(·) represents the emotion parsing module, which is divided into two parts: the multi-emotion classification module Mutiemoclassifier(·) and the temporal location module Temlocation(·).

[0091] Emo i = Mutiemoclassifier(M i )=MLP(Avgpool(M i ));

[0092] TMS i =Temlocation(M i )=Softmax(W c (M i )) (14);

[0093] In formula (14), MLP(·) represents a multi-layer perceptron, Avgpool(·) represents an average pooling operation, and Softmax is the activation function, that is, the corresponding activation layer, W c is a fully connected layer.

[0094] Step 6: Construct a total loss function based on the multi-emotion labels and time positioning information output in step 5, and back-propagate to adjust the trainable parameters in the time series multi-scale spatial dual model, thereby training the time series multi-scale spatial dual model.

[0095] Using formula (15), the multi-emotion labels Emo obtained by the temporal multi-scale spatial dual model during the i-th training i , time positioning information Tms i , and the corresponding real multi-emotion label results and time positioning information are combined, and the cross entropy error loss and mean square error loss functions are used to constrain the temporal multi-scale spatial dual model to locate the effect of multi-emotion perception positioning, and guide the optimization of the temporal multi-scale spatial dual model to improve the prediction accuracy:

[0096]

[0097] In formula (15), Emo i represents the multi-emotion label during the i-th training, Indicates the true emotional label, Tms i represents the time positioning information during the i-th training, Indicates the real time positioning information. represents the cross entropy loss function, Represents the mean square error loss function. In the two loss functions, the mathematical expression for a set of N training samples can be expressed as:

[0098]

[0099] Finally, determine the total loss function Loss(ξ t ) converges, that is, the loss value tends to be unchanged. If it converges, it means that the parameters at the i-th training are the optimal parameters, and this parameter is used as the temporal multi-scale spatial dual model for the final long-term speech multi-emotion perception and positioning. Otherwise, the i-th gradient back propagation is continued, and the parameters are updated, and the model parameters at the i+1-th training are obtained, i+1 is assigned to i, and the execution returns to step 2 in sequence until the temporal multi-scale spatial dual model converges.

[0100] Through steps one to six, this embodiment combines the low-dimensional time series SSD architecture module, the medium-dimensional time series SSD architecture module, the high-dimensional time series SSD architecture module and the emotion analysis module to realize the multi-emotion perception and positioning of long-term speech sequences from the perspective of time series multi-scale. This unique architectural design enables the time series multi-scale spatial dual model to extract rich long-term speech emotional information from multiple scales, effectively capture long-term and short-term time information, enhance the accuracy and robustness of multi-emotion recognition, and innovatively propose the new task of multi-emotion perception and positioning of long-term speech. In addition, the introduced state-space dual model effectively solves the performance bottleneck of traditional deep learning models in processing long-term speech through its characteristic of selectively forgetting in long-term multi-scale to retain key information, improves the computational efficiency of the model, and provides the possibility for realizing the application of multi-emotion perception and positioning of long-term speech sequences.

[0101] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A multi-emotion perception and localization method for long-duration speech, characterized in that: Inputting the long-duration speech to be recognized into the trained time-series multi-scale spatial dual model, and outputting multiple emotional states in the long-duration speech; The training process of the temporal multiscale spatial dual model is as follows: Step 1: Obtain long-term speech of multiple emotions, and extract them into initial long-term speech feature vectors respectively through a general speech pre-training model to form a training data set; Step 2: normalize the initial long-term speech feature vector and input it into the low-dimensional time series SSD architecture module to obtain a low-dimensional state space dual output; Step 3: Perform temporal downsampling and layer normalization operations on the low-dimensional state space dual output along the time dimension, and input the features reduced by half in the time dimension into the medium-dimensional temporal SSD architecture module to obtain the medium-dimensional state space dual output; Step 4: Perform time-series downsampling and layer normalization operations on the medium-dimensional state space dual output along its time dimension, and input the features reduced by half in the time dimension into the high-dimensional time series SSD architecture module to obtain the high-dimensional state space dual output; Step 5: perform feature fusion operation on the low-dimensional state space dual output, the medium-dimensional state space dual output and the high-dimensional state space dual output, and input the fused complex long-term multi-emotion speech time series features into the emotion analysis module to predict the final multi-emotion labels and time positioning information; Step 6: Construct a total loss function based on the multi-emotion labels and time positioning information output in step 5, and back-propagate to adjust the trainable parameters in the time series multi-scale spatial dual model, thereby training the time series multi-scale spatial dual model.

2. The multi-emotion perception and positioning method for long-term speech according to claim 1 is characterized in that: In steps 2 to 5, the structures of the low-dimensional time series SSD architecture module, the medium-dimensional time series SSD architecture module, and the high-dimensional time series SSD architecture module are consistent, all of which include a gating mechanism, a one-dimensional convolutional layer, an activation layer, and a state-space dual model. The time series SSD architecture module processes the input features as follows: The input features pass through the fully connected layer as the input of the state-space dual model; The input features are sequentially passed through the fully connected layer, the one-dimensional convolutional layer, and the activation layer as the input of the state-space dual model; The state-space dual output is obtained by multiplying the output of the input feature through the gating mechanism with the output of the state-space dual model.

3. The multi-emotion perception and positioning method for long-term speech according to claim 2 is characterized in that: The calculation formula of the state space dual model for the input features is as follows: s t =A t s t-1 +B t x t y t =C t s t Among them, s t ,s t-1 Represent the hidden state vector at time t and t-1 respectively, A t ,B t ,C t are the matrix parameters that change with time t, x t ,y t are the input features and output features of the state space dual model at time t respectively; Among them, the state space dual model is A t Set: Set all diagonal elements to the same value and all off-diagonal elements to zero.

4. The multi-emotion perception and positioning method for long-term speech according to claim 2 is characterized in that: The processing process of the gating mechanism on the input features is as follows: the output of the input features after passing through the fully connected layer and the activation layer in sequence is used as the output of the gating mechanism.

5. The multi-emotion perception and positioning method for long-term speech according to claim 1, characterized in that: In step five, the low-dimensional state space dual output, the medium-dimensional state space dual output and the high-dimensional state space dual output are subjected to feature fusion operation to obtain the complex long-term multi-emotion speech time series features, specifically: Perform linear interpolation on the dual output of the medium-dimensional state space and the dual output of the high-dimensional state space respectively, so as to restore them to the same dimension as the dual output of the low-dimensional state space in the time dimension; The low-dimensional state space dual output, the medium-dimensional state space dual output after linear interpolation, and the high-dimensional state space dual output after linear interpolation are concatenated and subjected to a linear layer dimensionality reduction operation to output complex long-term multi-emotional speech time series features.

6. The multi-emotion perception and positioning method for long-term speech according to claim 1, characterized in that: In step 5, the emotion analysis module includes a multi-emotion classification module and a time positioning module. The processing process of the emotion analysis module is as follows: The complex long-term multi-emotion speech time series features are input into the multi-emotion classification module, and then the multi-emotion labels are output after average pooling and multi-layer perceptron operation. The complex, long-term, and multi-emotional speech temporal features are input into the time localization module, and are operated through the fully connected layer and the activation layer to output the time localization information.

7. The multi-emotion perception and positioning method for long-term speech according to claim 6 is characterized in that: In step six, the total loss function is constructed as: Construct a cross entropy loss function based on the predicted output multiple emotion labels and the true emotion labels; Construct a mean square error loss function based on the predicted output time location information and the actual time location information; The total loss function is constructed based on the sum of the cross entropy loss function and the mean square error loss function.

8. A multi-emotion perception and positioning system for long-term speech, characterized in that: Inputting the long-duration speech to be recognized into the trained time-series multi-scale spatial dual model, and outputting multiple emotional states in the long-duration speech; The training process of the temporal multiscale spatial dual model is as follows: Step 1: Obtain long-term speech of multiple emotions, and extract them into initial long-term speech feature vectors respectively through a general speech pre-training model to form a training data set; Step 2: normalize the initial long-term speech feature vector and input it into the low-dimensional time series SSD architecture module to obtain a low-dimensional state space dual output; Step 3: Perform temporal downsampling and layer normalization operations on the low-dimensional state space dual output along the time dimension, and input the features reduced by half in the time dimension into the medium-dimensional temporal SSD architecture module to obtain the medium-dimensional state space dual output; Step 4: Perform time-series downsampling and layer normalization operations on the medium-dimensional state space dual output along its time dimension, and input the features reduced by half in the time dimension into the high-dimensional time series SSD architecture module to obtain the high-dimensional state space dual output; Step 5: perform feature fusion operation on the low-dimensional state space dual output, the medium-dimensional state space dual output and the high-dimensional state space dual output to obtain complex long-term multi-emotion speech time series features, and input the complex long-term multi-emotion speech time series features into the emotion analysis module to predict the final multi-emotion labels and time positioning information; Step 6: Construct a total loss function based on the multi-emotion labels and time positioning information output in step 5, and back-propagate to adjust the trainable parameters in the time series multi-scale spatial dual model, thereby training the time series multi-scale spatial dual model.

Images