A speech content emotion analysis method and system based on deep learning

Abstract

The application discloses a speech content emotion analysis method and system based on deep learning, and belongs to the technical field of speech signal processing. The method comprises the following steps: preprocessing an original speech signal to construct an acoustic feature sequence, and synchronously converting the acoustic feature sequence into a text sequence; extracting a deep text semantic feature vector and a multi-scale acoustic physical representation feature vector; realizing cross-modal nonlinear alignment through an energy constraint-based dynamic time warping algorithm, calculating a modal logic inconsistency degree, generating a conflict suppression factor, combining acoustic modal trustworthiness and text modal trustworthiness to dynamically evolve adaptive fusion weights, and generating an enhanced emotion feature matrix; anchoring a long sequence emotion keynote by a global context memory unit, outputting a global emotion representation vector, and performing emotion classification. Through nonlinear alignment, logic conflict correction and modal trust perception fusion, the application improves the accuracy of speech emotion analysis under complex contexts.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of speech signal processing and natural language processing, specifically relating to a method and system for sentiment analysis of speech content based on deep learning. Background Technology

[0002] With the iterative evolution of artificial intelligence technology, human-computer interaction has gradually transitioned from early rule-driven instruction to a stage of emotional and semantic interaction with cognitive intelligence. Voice sentiment analysis has thus become an interdisciplinary research field encompassing natural language processing, speech signal processing, and affective computing. The emotional information carried in speech has clear application value in scenarios such as intelligent customer service quality monitoring, digital assessment of mental health, fatigue detection in in-vehicle intelligent cockpits, and anthropomorphic interaction with virtual digital humans. Utilizing deep learning models to automatically learn emotional representations and determine tendencies in speech signals is one of the core technological paths to improve the intelligence level of human-computer interaction systems.

[0003] The invention patent scheme with publication number CN120279949A utilizes short-time Fourier transform to preprocess the original audio for environmental noise reduction, and combines linear prediction cepstral coefficients to extract voiceprint features. It then uses a multilayer perceptron model to nonlinearly map acoustic features such as speech rate, intonation, and energy distribution to infer the speaker's emotional state. This type of method equates emotion analysis with pattern recognition of audio signals, failing to fully utilize the linguistic properties of speech as an information carrier.

[0004] Patent CN106503805B proposes a machine learning-based bimodal human-to-human dialogue sentiment analysis method. It extracts deep word-level and sentence-level features from the text, linearly concatenates them with acoustic features, and then inputs the result into a classification model. Patent CN112560503B discloses a semantic sentiment analysis method that integrates deep features and temporal models. It utilizes convolutional neural networks to extract local semantic sentiment features, bidirectional long short-term memory networks to extract contextual semantic sentiment features, and then performs weighted fusion through an attention mechanism. Patent CN110059188B proposes a Chinese sentiment analysis method based on bidirectional temporal convolutional networks, using multi-layer dilated causal convolutions to extract forward and backward features from text sequences.

[0005] Acoustic features are extracted frame by frame, with a typical frame shift of 10ms, generating approximately 100 feature vectors per second. Text semantics are extracted word by word, with a speech rate of approximately 2 to 3 words per second. The two modalities differ by more than thirty orders of magnitude in terms of time scale. Existing techniques often employ linear concatenation or equal-length sampling for alignment, failing to consider the non-stationary nature of speech signals and the non-linear temporal shifts caused by speech rate fluctuations. This results in a mismatch between semantic center of gravity and acoustic stress on the time axis, leading to systematic biases in cross-modal attention calculations.

[0006] In ironic, implicit, or insincere contexts, a speaker's acoustic features (such as a high-pitched tone) may directly conflict with their true semantic core (such as a negative evaluation). Current technologies assume that the acoustic and textual modalities are consistent in their emotional expression, lacking mechanisms for detecting and correcting logical conflicts between modalities. When the emotional information conveyed by the two modalities is significantly inconsistent, simple feature splicing or weighted fusion can lead to mutual contamination of conflicting information, causing the final emotion judgment result to deviate from the true intent.

[0007] In real-world applications, ambient background noise can severely reduce the signal-to-noise ratio of acoustic features. Factors such as the speaker's accent, speech rate, and volume can also cause recognition errors in speech recognition engines, thereby reducing the semantic fidelity of text modalities. Existing technologies employ fixed fusion weights for all input samples or rely solely on end-to-end self-learning to determine weights, lacking real-time dynamic perception and robustness to disturbances regarding modal input quality. When the quality of a particular modality deteriorates significantly, fixed fusion strategies cannot promptly suppress the contributions of unreliable modalities, leading to contamination of the fused features.

[0008] Existing speech emotion analysis technologies have significant shortcomings in areas such as cross-modal nonlinear temporal alignment, modal logic conflict detection and correction, and adaptive perception of modal quality. Summary of the Invention

[0009] The purpose of this invention is to provide a speech content sentiment analysis method and system based on deep learning, which solves the technical problems of insufficient environmental adaptability caused by cross-modal temporal nonlinear mapping deviation, unresolved modal logic conflicts, and fixed modal quality assumptions in existing speech sentiment analysis technologies.

[0010] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A speech content sentiment analysis method based on modal reliability assessment and cross-modal nonlinear alignment includes the following steps: Step S1: Obtain the original speech signal to be analyzed, preprocess the original speech signal to construct an acoustic feature sequence, and simultaneously use a speech recognition engine to convert the original speech signal into a text sequence. Step S2: Construct a deep semantic enhancement model, input the text sequence into the deep semantic enhancement model for multi-level feature mapping, and extract deep text semantic feature vectors containing contextual logical relationships; Step S3: Construct a multi-scale acoustic spatiotemporal feature extraction model. Through a parallel temporal convolutional network and a recurrent neural network with a self-attention mechanism, perform joint feature extraction in the time and frequency domains of the acoustic feature sequence to generate an acoustic physical representation feature vector. Step S4: Establish a cross-modal nonlinear alignment and logical conflict correction mechanism: Using a dynamic time warping algorithm based on energy distribution constraints, find the optimal nonlinear mapping path between the acoustic physical representation feature vector and the deep text semantic feature vector to achieve precise alignment between the acoustic modal sequence and the text modal sequence; calculate the relative entropy between the sentiment distribution of the acoustic modal sequence and the sentiment distribution of the text modal sequence to obtain the modal logical inconsistency; when the modal logical inconsistency exceeds a preset threshold, generate a logical conflict suppression factor to correct the fusion weights; simultaneously, calculate the acoustic modal confidence based on the signal-to-noise ratio of the original speech signal and the energy distribution of the acoustic feature sequence, calculate the text modal confidence based on the word-level confidence output by the speech recognition engine, and dynamically calculate adaptive fusion weights based on the acoustic modal confidence, the text modal confidence, and the logical conflict suppression factor to generate an enhanced sentiment feature matrix; Step S5: Construct a long-sequence emotion evolution modeling layer based on global tone anchoring. Input the enhanced emotion feature matrix into the combined architecture of bidirectional long short-term memory network and global context memory unit. Extract the global emotion tone vector of the sequence through global average pooling. Adaptively fuse the global emotion tone vector with the instantaneous hidden state of the bidirectional long short-term memory network using a gating mechanism to capture the dynamic fluctuation trajectory of the emotion state over time and output the global emotion representation vector. Step S6: Perform nonlinear classification mapping on the global sentiment representation vector using an end-to-end sentiment classifier to output the final sentiment analysis result.

[0011] Furthermore, in step S1, the preprocessing step of the original speech signal specifically includes: Step S11: Perform pre-weighting processing on the mono original speech signal with a sampling frequency of 16kHz and a sampling bit depth of 16bit, and set the pre-weighting coefficient to 0.97. Step S12: The pre-weighted signal is framed using a Hamming window, with a frame length of 25ms and a frame shift of 10ms, ensuring an inter-frame overlap rate of 50%. Step S13: Perform a fast Fourier transform on each frame of signal, and map the transformed signal to a Mel scale filter bank to extract 40-dimensional Mel frequency cepstral coefficients and their first-order and second-order difference features, forming a 120-dimensional acoustic feature sequence.

[0012] Furthermore, in step S2, the deep semantic enhancement model adopts a 12-layer bidirectional transformation encoder structure, with the hidden layer dimension set to 768 and the number of heads for multi-head attention set to 12; the deep semantic enhancement model internally captures long-distance semantic dependencies in the text content through a multi-head self-attention mechanism.

[0013] Furthermore, in step S3, the operation flow of the multi-scale acoustic spatiotemporal feature extraction model includes: Step S31: Input the acoustic feature sequence into a temporal convolutional network consisting of four layers of dilated convolutions, with the dilation factor of each layer increasing in a power of two order. Step S32: Simultaneously input the acoustic feature sequence into the recurrent neural network, and use the gated recurrent unit to perform temporal modeling of the pitch fluctuations of the speech signal; Step S33: The output of the temporal convolutional network and the output of the recurrent neural network are aggregated through a weighted fusion layer to generate the acoustic physical representation feature vector with consistent dimensions.

[0014] Furthermore, in step S4, the specific method for finding the optimal nonlinear mapping path using the dynamic time warping algorithm based on energy distribution constraints is as follows: The alignment cost is calculated as follows: the square of the Euclidean distance between the acoustic feature vector after feature mapping and the word embedding vector after mapping is added to the product of the energy constraint weight coefficient and the absolute value of the energy difference; wherein, the absolute value of the energy difference is the absolute value of the difference between the short-time energy of the acoustic feature of the frame and the average energy reference value of the time interval corresponding to the word.

[0015] Furthermore, the optimal nonlinear mapping path is found recursively to find the path with the minimum cumulative distance. The specific recursive logic is as follows: the cumulative distance of a grid point is equal to the alignment cost of the current grid point, plus the minimum cumulative distance among its three adjacent grid points to the left, below, and to the lower left. After the calculation is completed, the optimal alignment path that minimizes the total cumulative distance is obtained by backtracking.

[0016] Furthermore, in step S4, the modal logic inconsistency is calculated as follows: the aligned acoustic modal sequence and text modal sequence are input into the sentiment probing head, and the acoustic sentiment probability distribution and text sentiment probability distribution are output; the sum of the bidirectional Kohlbek-Leibler divergence between the two probability distributions is calculated, and the sum is mapped to the zero-to-one interval through a logistic regression function as the modal logic inconsistency.

[0017] Furthermore, in step S4, the logic for generating the logic conflict suppression factor is as follows: when the modal logic inconsistency exceeds a preset threshold, the logic conflict suppression factor is equal to one minus the product of the conflict sensitivity coefficient and the inconsistency exceeding the threshold; when the modal logic inconsistency does not exceed the preset threshold, the logic conflict suppression factor is equal to one.

[0018] Furthermore, in step S4, the calculation logic for the acoustic modality confidence is as follows: the normalized signal-to-noise ratio weight is weighted and summed with the negative exponential quality score weight of the acoustic energy sequence variance; the calculation logic for the text modality confidence is as follows: the arithmetic mean of the recognition confidence scores of all words in the text sequence output by the speech recognition engine is calculated.

[0019] Furthermore, in step S4, the adaptive fusion weight is calculated as follows: the fusion weight of the acoustic modality is equal to the product of the acoustic modality confidence level and the logical conflict suppression factor, divided by the sum of the product and the text modality confidence level; the fusion weight of the text modality is equal to one minus the fusion weight of the acoustic modality.

[0020] Furthermore, in step S5, the adaptive fusion logic of the gating mechanism is as follows: first, the instantaneous hidden state is concatenated with the global sentiment tone vector, and a gating scalar is generated through a mapping function and a logistic regression function; the fused hidden state is equal to the element-wise product of the original hidden state minus the gating scalar, plus the element-wise product of the gating scalar and the global sentiment tone vector after linear mapping.

[0021] Furthermore, in step S6, the sentiment classifier is optimized using a hierarchical contrastive learning joint loss function, the total loss function of which is equal to the weighted sum of the cross-entropy loss function, the sentence-level cohesion loss function, and the sequence-level smoothing loss function.

[0022] Furthermore, the calculation logic of the sentence-level cohesion loss function is as follows: within the training batch, using the global sentiment representation vector of the current sample as the anchor point, calculate the cosine similarity normalized log-likelihood with all positive sentiment label samples; the calculation logic of the sequence-level smoothing loss function is as follows: for the sentiment category prediction probability distribution vectors of adjacent time steps in a long speech sequence, calculate the square of their Euclidean distance.

[0023] On the other hand, the present invention also discloses a deep learning-based speech content sentiment analysis system for performing a deep learning-based speech content sentiment analysis method as described above, characterized in that it includes: The data acquisition and preprocessing module is used to acquire the raw speech signal and perform preprocessing, extract acoustic feature sequences, and generate the corresponding text sequences; The semantic enhancement extraction module is used to perform deep semantic mapping on the text sequence and output a deep text semantic feature vector. A multi-scale acoustic feature module is used to extract acoustic physical representation feature vectors from the acoustic feature sequence; The cross-modal nonlinear alignment and fusion module is used to establish a nonlinear alignment mechanism based on energy distribution constraints, and combines the inhibition factor generated by modal trust assessment and logical conflict detection to dynamically calculate the fusion weight and output the enhanced sentiment feature matrix. The global tone anchoring evolution module is used to extract the global sentiment tone vector using global average pooling and to fuse it with the hidden state through a gating mechanism to capture the dynamic evolution of sentiment. The Emotion Decision Classification module is used to map the global emotion representation vector to a specific emotion category and output the result.

[0024] Furthermore, the system also includes an adaptive noise suppression submodule, which is located at the front end of the data acquisition and preprocessing module and uses spectral subtraction based on deep residual networks to suppress environmental background noise.

[0025] Furthermore, in step S4, the step of finding the optimal nonlinear mapping path using a dynamic time warping algorithm based on energy distribution constraints further includes: Acquiring acoustic modal trust and frame-level instantaneous signal-to-noise ratio The energy constraint weights acting on the alignment cost matrix are dynamically calculated through an adaptive mapping function. The adaptive mapping function makes the energy constraint weights approach zero when the acoustic modal confidence is lower than a preset threshold, and enhances the alignment constraint contribution of the absolute value of the energy difference term in a high signal-to-noise ratio environment. The alignment cost calculation formula is revised as follows: ,in This is the effective speech energy estimation after preprocessing by spectral subtraction.

[0026] Furthermore, the logic for generating the logic conflict suppression factor in step S4 further includes: Calculate the average normalized cumulative distance of the optimal alignment path in dynamic time warping. And generate alignment confidence accordingly. ; The logical conflict suppression factor is weighted and modified based on the alignment confidence level to generate an enhanced logical conflict suppression factor. ; When the alignment confidence level approaches 1, even with high modal logic inconsistency, the enhanced logic conflict suppression factor remains close to 1 to preserve the deliberately acoustically modified emotional characteristics.

[0027] Furthermore, in step S4, the calculation method for the alignment cost further introduces a micro-prosodic tension index to compensate for the energy characteristic attenuation under inhibitory emotional expression. The specific calculation logic includes: Extract the fundamental frequency perturbation sequence and spectral flux sequence of acoustic features from each frame. Then, use a nonlinear mapping function to perform feature concatenation and normalization on the fundamental frequency perturbation sequence and the spectral flux sequence to generate a micro-prosodic tension index. ; Dynamically monitor the short-time energy of acoustic features in each frame. When the short-time energy Below the preset active energy threshold And the micro-rhythmic tension index The tension trigger threshold is higher than the preset threshold. At that time, the energy constraint compensation mechanism is activated; At the time point when the energy constraint compensation mechanism is activated, the absolute value of the energy difference in the alignment cost calculation formula is configured to be related to the natural exponential function. The form of multiplication, where This is the tension amplification factor; the modified alignment cost calculation formula is as follows: This causes the mapping path of the dynamic time warping algorithm to tilt towards the high-tension physical frame under low sound intensity conditions.

[0028] In step S4, the calculation of the modal logical inconsistency degree is superimposed on the local sentiment gradient features of the time dimension, and the specific calculation rules include: Following the time step sequence, first-order discrete difference operations are performed on the acoustic sentiment tendency probability distribution and the text sentiment tendency probability distribution respectively to extract the acoustic local sentiment evolution gradient vector. and the local sentiment evolution gradient vector of the text ; Calculate the acoustic local emotion evolution gradient vector With the local sentiment evolution gradient vector of the text The cosine distance between them is used to construct a local dynamic conflict penalty sequence; The peak extrema in the local dynamic conflict penalty sequence are extracted by a sliding time window, and the peak extrema are multiplied by a preset evolution penalty coefficient. Generate dynamic divergence gain term; The dynamic divergence gain term is superimposed on the sum of the bidirectional Kohlbek-Leibler divergences, and the logistic regression function is used for nonlinear compression to output the updated modal logical inconsistency, so as to capture the local emotional mutations that occur in the acoustic modality under the text modal semantic stationary state.

[0029] Compared with the prior art, the present invention has the following beneficial effects: This invention employs a dynamic time warping algorithm based on energy distribution constraints. By incorporating constraints such as the short-time energy of acoustic frames and the average energy reference values ​​of corresponding word intervals into the cost matrix, it achieves precise alignment of acoustic modal sequences and text modal sequences on a nonlinear time axis. This alignment mechanism helps eliminate the mismatch between semantic center of gravity and acoustic stress caused by speech rate fluctuations and the non-stationary characteristics of speech signals.

[0030] This invention maps aligned acoustic and textual modalities to sentiment probability spaces, calculates the bidirectional KL divergence between the two distributions as a quantitative indicator of modal logical inconsistency, and generates a logical conflict suppression factor when the inconsistency exceeds a preset threshold, dynamically adjusting the fusion weights of the acoustic modalities. This mechanism provides an effective means of handling modal logical conflict scenarios such as insincerity and irony.

[0031] This invention constructs acoustic modality trust and text modality trust, and combines them with a logical conflict suppression factor to achieve a three-factor dynamic evolution of the fusion weight. This mechanism enables the system to automatically suppress the contribution of the acoustic modality in a noisy environment, automatically reduce the influence of the text modality when the speech recognition confidence is low, and automatically bias towards text semantics when logical conflicts are detected, thereby improving the system's dual adaptability to environmental interference and contextual conflicts.

[0032] This invention introduces a global contextual memory unit on top of a bidirectional long short-term memory network. It extracts the global sentiment tone of the entire speech segment through global average pooling and adaptively fuses the global sentiment tone with the transient hidden state using a gating mechanism. This mechanism helps maintain the logical coherence and consistency of sentiment analysis results in long dialogues. Attached Figure Description

[0033] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0034] Figure 1 This is a flowchart of the cross-modal nonlinear alignment and logic conflict correction mechanism of the present invention.

[0035] Figure 2 This is a flowchart of the long-sequence emotion evolution modeling process that anchors the global tone of this invention.

[0036] Figure 3 This is an overall flowchart of the method described in this invention. Detailed Implementation

[0037] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the embodiments of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0038] The following is in conjunction with the appendix Figures 1-3 The embodiments of the present invention will be described in detail below.

[0039] This invention provides a deep learning-based method for sentiment analysis of speech content, comprising the following steps: Step S1: Obtain the original speech signal to be analyzed, preprocess the original speech signal to construct an acoustic feature sequence, and simultaneously use a speech recognition engine to convert the original speech signal into a text sequence.

[0040] Step S2: Construct a deep semantic enhancement model. Input the text sequence into the deep semantic enhancement model for multi-level feature mapping and extract deep text semantic feature vectors containing contextual logical relationships.

[0041] Step S3: Construct a multi-scale acoustic spatiotemporal feature extraction model. Through a parallel temporal convolutional network and a recurrent neural network with a self-attention mechanism, perform joint feature extraction in the time and frequency domains of the acoustic feature sequence to generate an acoustic physical representation feature vector.

[0042] Step S4: Establish a cross-modal nonlinear alignment and logical conflict correction mechanism. Using a dynamic time warping algorithm based on energy distribution constraints, find the optimal nonlinear mapping path between the acoustic physical representation feature vector and the deep text semantic feature vector to achieve precise alignment between the acoustic modal sequence and the text modal sequence. Calculate the relative entropy between the sentiment distribution of the acoustic modal sequence and the sentiment distribution of the text modal sequence to obtain the modal logical inconsistency. When the modal logical inconsistency exceeds a preset threshold, generate a logical conflict suppression factor to correct the fusion weights. Simultaneously, calculate the acoustic modal confidence based on the signal-to-noise ratio of the original speech signal and the energy distribution of the acoustic feature sequence, calculate the text modal confidence based on the word-level confidence output by the speech recognition engine, and dynamically calculate adaptive fusion weights based on the acoustic modal confidence, the text modal confidence, and the logical conflict suppression factor to generate an enhanced sentiment feature matrix.

[0043] Step S5: Construct a long-sequence emotion evolution modeling layer based on global tone anchoring. Input the enhanced emotion feature matrix into the combined architecture of bidirectional long short-term memory network and global context memory unit. Extract the global emotion tone of the sequence through global average pooling. Adaptively fuse the global emotion tone with the instantaneous hidden state of the bidirectional long short-term memory network using a gating mechanism. Capture the dynamic fluctuation trajectory of the emotion state over time and output the global emotion representation vector.

[0044] Step S6: Perform nonlinear classification mapping on the global sentiment representation vector using an end-to-end sentiment classifier to output the final sentiment analysis result.

[0045] In a preferred embodiment of the present invention, step S1, the step of preprocessing the original speech signal, specifically includes: Step S11: Perform pre-weighting processing on the mono original speech signal with a sampling frequency of 16kHz and a sampling bit depth of 16bit. The pre-weighting coefficient is set to 0.97 to compensate for the drop in the high-frequency part of the speech signal.

[0046] Step S12: Use a Hamming window to perform frame division processing on the pre-weighted signal, set the frame length to 25ms and the frame shift to 10ms, to ensure that there is a 50% overlap area between frames.

[0047] Step S13: Perform a fast Fourier transform on each frame of signal, and map the transformed signal to a Mel scale filter bank to extract 40-dimensional Mel frequency cepstral coefficients and their first-order and second-order difference features, forming a 120-dimensional acoustic feature sequence.

[0048] In a preferred embodiment of the present invention, in step S2, the deep semantic enhancement model adopts a multi-layer bidirectional transformation encoder structure, the internal calculation logic of which includes: Step S21: Map each word in the text sequence to a fixed-dimensional embedding vector and introduce positional encoding vectors to preserve word order information.

[0049] Step S22 involves performing deep semantic abstraction on the embedded vector using 12 stacked encoding layers. Each encoding layer contains a multi-head self-attention sub-layer and a position-wise feedforward neural network sub-layer. The multi-head self-attention sub-layer obtains the query matrix, key matrix, and value matrix by linearly transforming the input sequence, and uses scaling dot product operations to capture long-distance semantic dependencies in the text content, outputting the deep text semantic feature vector.

[0050] In a preferred embodiment of the present invention, the operation flow of the multi-scale acoustic spatiotemporal feature extraction model in step S3 is as follows: Step S31: The acoustic feature sequence is input into a temporal convolutional network consisting of four layers of dilated convolutions. The dilation factor of each layer increases in a power of two order to expand the receptive field layer by layer and extract local sound intensity, speech rate and spectral envelope fluctuation features.

[0051] Step S32: The acoustic feature sequence is synchronously input into the recurrent neural network, and the pitch fluctuation of the speech signal is modeled in time using a gated recurrent unit.

[0052] Step S33: The output of the temporal convolutional network and the output of the recurrent neural network are aggregated through a weighted fusion layer to generate acoustic physical representation feature vectors with consistent dimensions, ensuring that the model can simultaneously cover the microscopic transient features and macroscopic temporal features of acoustic signals.

[0053] In a preferred embodiment of the present invention, the specific process of establishing the cross-modal nonlinear alignment and logic conflict correction mechanism in step S4 includes: Step S41: Define the acoustic physical representation feature vector as an acoustic modal sequence and the deep text semantic feature vector as a text modal sequence.

[0054] Step S42: Find the optimal nonlinear mapping path between the acoustic modal sequence and the text modal sequence using a dynamic time warping algorithm based on energy distribution constraints.

[0055] Define the acoustic modal sequence as A, containing n frames of feature vectors; and the text modal sequence as T, containing m word embedding vectors. Construct a cost matrix D, where each element d(i,j) represents the alignment cost between the i-th acoustic frame and the j-th text word. The formula for calculating the alignment cost is:

[0056] in, This represents the acoustic feature vector of the i-th frame; Represents the embedding vector of the j-th word; The acoustic feature mapping function is represented by a single-layer fully connected network. The text feature mapping function is implemented using a single-layer fully connected network. This represents the square of the Euclidean distance between vectors; The short-time energy of the i-th frame is obtained by summing the squares of the signal amplitudes of that frame and taking the logarithm. The average energy reference value for the time interval corresponding to the j-th word is calculated from the acoustic signal based on the time label provided by the speech recognition engine. This represents the energy constraint weighting coefficient, which is set to 0.15 in this embodiment.

[0057] Find the path with the minimum cumulative distance from the starting point to the ending point using a recursive approach. (Cumulative distance) The recursive formula is:

[0058] in, This represents taking the minimum cumulative distance between three adjacent grid points. After dynamic programming calculation, the optimal alignment path is obtained through backtracking. This optimal alignment path maps each frame in the acoustic modal sequence to the most relevant word position in the text modal sequence in a non-linear manner.

[0059] Step S43: Input the aligned acoustic modal sequence and text modal sequence into the pre-constructed sentiment probing head, and output the acoustic sentiment probability distribution. and the probability distribution of text sentiment Calculate the modal logic inconsistency degree between two distributions. The calculation formula is:

[0060] in, This represents the KL divergence between two probability distributions; This represents the sigmoid mapping function, which maps the sum of KL divergences to the interval between 0 and 1. When Exceeding the preset threshold When the current audio segment is determined to be a "disingenuous" scenario, a logic conflict correction mechanism is triggered. In this embodiment, a preset threshold is used. Set to 0.6.

[0061] Step S44, the specific implementation method of the logical conflict correction mechanism is as follows: define a logical conflict suppression factor. .when hour, The calculation formula is:

[0062] in, This represents the conflict sensitivity coefficient, which is set to 0.8 in this embodiment. When hour, This logical conflict suppression factor will directly affect the fusion weights of the acoustic modes.

[0063] Step S45: Calculate the acoustic modal confidence level. and text modality trust Acoustic modal confidence The calculation formula is:

[0064] in, The signal-to-noise ratio (SNR) of the original speech signal is calculated after separating speech segments from silence segments through speech activity detection. This represents a normalization function that maps decibel values ​​to the interval between 0 and 1. Represents acoustic energy sequence The variance of the background noise is used to characterize its stationarity. Map the energy variance to a quality score between 0 and 1; and These represent weighting coefficients, which are set to 0.6 and 0.4 respectively in this embodiment.

[0065] Text modality trust The calculation formula is:

[0066] in, This represents the confidence score of the speech recognition engine for the i-th word output, with a value ranging from 0 to 1; This represents the total number of words in the text sequence.

[0067] Step S46, based on acoustic modal confidence Text modality trust and logical conflict suppression factor Dynamically calculate adaptive fusion weights. Adaptive fusion weights for acoustic modes. The calculation formula is:

[0068] Adaptive fusion weights for text modalities The calculation formula is:

[0069] Step S47, according to the adaptive fusion weights and For acoustic modal eigenvectors and text modal feature vectors Weighted fusion is performed, and an enhanced sentiment feature matrix is ​​generated through residual connections and layer normalization operations.

[0070] In a preferred embodiment of the present invention, the operating mechanism of the long-sequence emotion evolution modeling layer based on global tone anchoring in step S5 is as follows: Step S51: Input the enhanced emotion feature matrix into the bidirectional long short-term memory network and use the concatenation of forward hidden state and backward hidden state to capture the bidirectional emotion evolution information of the context.

[0071] Step S52: Set up an independent global context memory unit, compress the enhanced sentiment feature matrix along the time dimension through a global average pooling layer, and extract the global sentiment tone vector of the sequence. Global sentiment vector The calculation formula is:

[0072] in, The first element of the enhanced sentiment feature matrix represents the... row feature vector, Indicates the number of audio frames.

[0073] Step S53, at each time step The global sentiment tone vector is controlled by a gating mechanism. Hidden states of bidirectional long short-term memory networks Perform adaptive fusion. Gated scalar The calculation formula is:

[0074] in, This indicates that the bidirectional long short-term memory network is at time 10:00. The hidden state; Represents the global sentiment tone vector; This represents a vector concatenation operation; Represents the weight matrix; Indicates the bias term; Represents the sigmoid function. The hidden state after fusion. The calculation formula is:

[0075] in, This represents a linear mapping matrix used to map the global sentiment tone vector. Mapped to Same dimensional space; This indicates element-wise multiplication.

[0076] In a preferred embodiment of the present invention, in step S6, the mapping process of the emotion classifier includes: Step S61: Input the global sentiment representation vector into the fully connected layer for high-dimensional dimensionality reduction mapping.

[0077] Step S62: Calculate the probability distribution of each emotion category using the Softmax function. The emotion categories include at least: anger, happiness, sadness, fear, surprise, disgust, and neutrality.

[0078] Step S63: Iteratively optimize the model using the hierarchical contrastive learning joint loss function. Total loss function. The calculation formula is:

[0079] in, Represents the cross-entropy loss function; This represents the sentence-level cohesion loss function. Represents the sequence-level smoothing loss function; and The balance coefficient is set to 0.1 in this embodiment.

[0080] Sentence-level cohesion loss function The calculation formula is:

[0081] in, Represents the training batch sample set; Indicates the relationship with the first A set of positive samples with the same sentiment label; Indicates that within the batch, excluding the first... The set of all samples outside of the given sample; Indicates the first The global sentiment representation vector of each sample; Represents the cosine similarity function; This represents the temperature over-parameter, which is set to 0.5 in this embodiment.

[0082] Sequence-level smoothing loss function The calculation formula is:

[0083] in, Indicates the number of segments into which a long speech sequence is divided; Indicates the first The probability distribution vector of sentiment categories at each time step; This represents the square of the Euclidean distance between vectors.

[0084] Secondly, the present invention also provides a speech content sentiment analysis system based on deep learning, comprising: The data acquisition and preprocessing module is used to acquire raw speech signals and perform pre-weighting, framing, and windowing processing, extract acoustic feature sequences, and integrate a speech recognition interface to generate corresponding text sequences.

[0085] The semantic enhancement extraction module integrates a multi-layer bidirectional transformation encoder to perform deep semantic mapping on the text sequence and output deep text semantic feature vectors.

[0086] The multi-scale acoustic feature module includes a parallel temporal convolutional network and a self-attention recursive network, used to extract acoustic physical representation feature vectors from the acoustic feature sequence.

[0087] A cross-modal nonlinear alignment and fusion module is used to establish a nonlinear alignment mechanism and a logical conflict correction mechanism between acoustic features and text features, generating an enhanced sentiment feature matrix. The cross-modal nonlinear alignment and fusion module includes: a temporal alignment unit, used to find the optimal nonlinear mapping path between acoustic modal sequences and text modal sequences using a dynamic time warping algorithm based on energy distribution constraints; a logical conflict detection unit, used to calculate the modal logical inconsistency degree between the acoustic sentiment tendency probability distribution and the text sentiment tendency probability distribution, and generate a logical conflict suppression factor when the modal logical inconsistency degree exceeds a preset threshold; a modal trust evaluation unit, used to calculate the acoustic modal trust degree based on the signal-to-noise ratio of the original speech signal and the energy distribution of the acoustic feature sequence, and calculate the text modal trust degree based on the word-level confidence degree output by the speech recognition engine; and an adaptive weight calculation unit, used to dynamically calculate the adaptive fusion weights of the acoustic modality and the text modality based on the acoustic modal trust degree, the text modal trust degree, and the logical conflict suppression factor.

[0088] The global tone anchoring evolution module consists of a bidirectional long short-term memory network and a global contextual memory unit. It is used to extract the global sentiment tone of the sequence through global average pooling, and adaptively fuse the global sentiment tone with the instantaneous hidden state of the bidirectional long short-term memory network using a gating mechanism to capture the long-sequence dynamic evolution law of sentiment and output the global sentiment representation vector.

[0089] The emotion decision classification module is used to map the global emotion representation vector to a specific emotion category and output the analysis results based on a preset emotion tag library.

[0090] In a preferred embodiment of the present invention, the system further includes an adaptive noise suppression submodule, which is located at the front end of the data acquisition and preprocessing module and uses spectral subtraction based on deep residual networks to suppress environmental background noise before feature extraction.

[0091] In a preferred embodiment of the present invention, the semantic enhancement extraction module uses an encoder with 12 layers, a hidden layer dimension of 768, and a multi-head attention head of 12.

[0092] To facilitate a better understanding of the present invention by those skilled in the art, the present invention will be further illustrated below with reference to specific implementation examples.

[0093] Example 1: This example discloses a speech content sentiment analysis method based on deep learning. In the specific implementation process, the analysis process begins with the digital acquisition of the original speech signal and the construction of preliminary features.

[0094] Step S1 aims to convert unstructured analog speech into a structured tensor that can be processed by a deep learning model, and simultaneously complete the initial transcription of semantic information. The system acquires the raw speech signal to be analyzed, which is limited to a sampling frequency of 16kHz and a sampling bit depth of 16bit during the acquisition phase. This parameter selection is based on a balance between acoustic sampling theorem and computational resources, covering the main frequency range of human speech while controlling the data throughput of subsequent processing.

[0095] In step S11, the system performs pre-weighting processing on the original mono audio signal. This process is implemented using a high-pass filter, with the pre-weighting coefficient set to 0.97. The purpose is to compensate for the high-frequency drops in the audio signal during radiation and improve the signal-to-noise ratio of the high-frequency formants.

[0096] In step S12, the pre-weighted signal enters the framing stage. The system uses a Hamming window for windowing processing, with a frame length of 25ms and a frame shift of 10ms. The 50% overlap setting can maintain the continuity of the speech signal over time and suppress spectral leakage caused by framing.

[0097] In step S13, the system performs a fast Fourier transform on each frame of signal to convert the time-domain signal into a frequency-domain signal.

[0098] The frequency domain signal is further mapped to 40 sets of Mel-scale filters. By performing logarithmic operations and discrete cosine transforms on the filter outputs, 40-dimensional Mel-frequency cepstral coefficients are extracted. The system further calculates and concatenates the first and second differences of the Mel-frequency cepstral coefficients to construct a 120-dimensional acoustic feature sequence. This sequence reflects the dynamic changes in the timbre, energy distribution, and spectral envelope of the speech. Simultaneously, the system invokes a speech recognition engine to convert the original speech signal into a text sequence with timestamp information and vocabulary-level confidence scores.

[0099] In step S2, a multi-layer bidirectional transformation encoder structure is used for deep semantic enhancement. In step S21, each word in the text sequence is mapped to a 768-dimensional embedding vector, and a positional encoding vector is introduced to preserve word order information. In step S22, the model is processed through 12 stacked encoding layers. Each encoding layer integrates 12 independent multi-head self-attention sublayers. This mechanism enables the model to spontaneously allocate attention to the context words that contribute the most to the current semantics when processing text, extracting long-distance semantic dependencies and outputting deep text semantic feature vectors.

[0100] Step S3: Construct a multi-scale acoustic spatiotemporal feature extraction model.

[0101] In step S31, the 120-dimensional acoustic feature sequence is fed into a temporal convolutional network consisting of four layers of dilated convolutions. The dilation factors of each layer increase in power-of-two increments, and the kernel size is 3. In step S32, the acoustic feature sequence is synchronously input into a gated recurrent unit, with the hidden layer dimension set to 256, to perform temporal modeling of the pitch fluctuations of the speech signal. In step S33, the local features extracted by the temporal convolutional network and the macroscopic temporal features extracted by the recurrent neural network are aggregated through a learnable weighted fusion layer to generate an acoustic physical representation feature vector.

[0102] Step S4 is the core of this invention, which establishes a cross-modal nonlinear alignment and logic conflict correction mechanism.

[0103] In step S41, the acoustic physical characterization feature vector is defined as an acoustic mode sequence. , dimension The deep text semantic feature vector is defined as a text modality sequence. , dimension .

[0104] In step S42, the optimal nonlinear mapping path is found using a dynamic time warping algorithm based on energy distribution constraints. The acoustic modal sequence is mapped to the same semantic space as the text modal sequence using a feature mapping function. via feature mapping function Mapping to the same semantic space. Constructing the cost matrix. ,element The calculation formula is:

[0105] in, For the first The short-time energy of a frame; For the first Average energy reference value for each word within a corresponding time interval; The value is set to 0.15. The introduction of the energy constraint term enables the dynamic time warping algorithm to consider both the semantic similarity of the feature space and the physical correspondence between acoustic energy and semantic centroid when searching for the optimal path.

[0106] The path with the minimum cumulative distance is found by recursively applying a formula, and the optimal alignment path is obtained by backtracking. This path maps each frame in the acoustic modal sequence to the most relevant word position in the text modal sequence in a non-linear manner.

[0107] In step S43, the aligned acoustic modal sequence and text modal sequence are input into the sentiment probe, and the acoustic sentiment probability distribution is output. and the probability distribution of text sentiment Calculate the modal logic inconsistency degree. .when Exceeding the threshold When this occurs, the logic conflict correction mechanism is triggered.

[0108] In step S44, a logical conflict suppression factor is defined. .

[0109] when hour, , Set to 0.8; when hour, . Fusion weights that directly affect acoustic modes.

[0110] In step S45, the acoustic modal confidence level is calculated. and text modality trust .

[0111] , , .in It is calculated by separating speech segments and silence segments through speech activity detection. Let be the variance of the acoustic energy sequence. Take the arithmetic mean of the confidence scores for all words.

[0112] In step S46, based on , and Dynamically calculate adaptive fusion weights. Adaptive fusion weights for acoustic modes. Text modality fusion weights When acoustic modal confidence decreases or a logical conflict is detected, Automatically decreases.

[0113] In step S47, according to and Acoustic modal eigenvectors and text modal feature vectors Weighted fusion is performed, and an enhanced sentiment feature matrix is ​​generated through residual connections and layer normalization operations. .

[0114] Step S5 involves long-sequence emotion evolution modeling based on global tone anchoring.

[0115] In step S51, the enhanced sentiment feature matrix The input is a bidirectional long short-term memory network, which captures the bidirectional emotional evolution information of the context by concatenating the forward and backward hidden states. In step S52, the global sentiment tone vector of the sequence is extracted by global average pooling. .

[0116] In step S53, a gating mechanism is used to... Hidden states of bidirectional long short-term memory networks Perform adaptive fusion. Gated scalar Hidden state after fusion When local emotional features are obvious and the confidence level is high, Smaller; when local emotional characteristics are vague or contradictory, The effect is significant, enhancing the overall emotional tone correction.

[0117] Step S6 outputs the final result through the sentiment classifier. Step S61 uses a fully connected layer to perform high-dimensional dimensionality reduction mapping.

[0118] Step S62 uses the Softmax function to output seven emotion probability distributions.

[0119] Step S63 utilizes hierarchical contrastive learning joint loss function to optimize the model, with the total loss function being... , . Narrowing the feature distance between samples of the same emotion category, Constrain the smooth evolution of sentiment prediction results at adjacent time steps.

[0120] To verify the technical effectiveness of the method and system described in this invention, a comparative experiment was conducted on the public multimodal emotion dataset IEMOCAP and real intelligent customer service recording data, totaling approximately 15,000 voice samples.

[0121] Comparative Example 1: Acoustic and textual dual-modal analysis was used to extract MFCC acoustic features and deep textual features, which were then linearly concatenated and classified using a support vector machine.

[0122] Comparative Example 2: For text modality analysis only, CNN is used to extract local semantic sentiment features, Bi-LSTM is used to extract contextual semantic sentiment features, and attention-weighted fusion is used for classification.

[0123] Comparative Example 3: For text modality analysis only, multi-layer dilated causal convolution is used to extract forward and backward features from the text sequence, and the features are concatenated and then classified.

[0124] Comparative Example 4: Based on Example 1, dynamic time regularization alignment was removed, and linear interpolation alignment was adopted.

[0125] Comparative Example 5: The logic conflict detection and correction mechanism was removed from Example 1.

[0126] Comparative Example 6: Based on Example 1, modal trust calculation was removed, and fixed fusion weights were used.

[0127] The evaluation metrics included weighted accuracy, unweighted accuracy, and accuracy in recognizing insincere statements. All experimental groups used the same dataset partitioning (7:1:2), batch size 32, Adam optimizer, initial learning rate 0.0001, maximum training epochs of 150, and an early stopping mechanism. Experimental data are shown in Table 1 below. Table 1:

[0128] Comparative Examples 1 to 3 are based on existing technical solutions. Comparative Example 1 has a weighted accuracy of 65.3%. Comparative Examples 2 and 3 only analyze the text modality, excluding the acoustic modality, with weighted accuracy rates of 68.7% and 69.2%, respectively. Data from Comparative Example 4 shows that after removing dynamic time-normalization alignment, the weighted accuracy decreases from 85.7% to 78.5%, and the accuracy for recognizing "insincere" scenes decreases from 83.1% to 67.2%. Data from Comparative Example 5 shows that after removing the logical conflict detection and correction mechanism, the accuracy for recognizing "insincere" scenes decreases from 83.1% to 61.5%. Data from Comparative Example 6 shows that after removing modal trust calculation, the weighted accuracy decreases from 85.7% to 80.2%. Example 1 outperforms all comparative examples in terms of weighted accuracy, unweighted accuracy, and accuracy for recognizing "insincere" scenes.

[0129] Example 2: This example describes the implementation of a dynamic time warping alignment mechanism. The acoustic modality extracts features frame by frame, with each frame spanning 25ms and a frame shift of 10ms, resulting in approximately 100 frames per second. The text modality extracts features word by word, approximately 2 to 3 words per second. The two modalities differ by orders of magnitude in their time scales.

[0130] In the coarse-grained alignment stage, the time label corresponding to each word output by the speech recognition engine is used to establish an initial correspondence between each word in the text sequence and the corresponding speech frame interval in the acoustic modality sequence. For example, the word "front end" corresponds to 0.0 to 0.5 seconds, which corresponds to the acoustic features of frames 1 to 50.

[0131] In the fine-grained alignment stage, based on the frame intervals established by coarse-grained alignment, a dynamic time warping algorithm is used for fine alignment. For the first... For each word, an acoustic frame interval is defined, a cost matrix is ​​constructed, and the optimal alignment path is calculated. The optimal correspondence weight between each frame and the word within that interval is obtained using a dynamic temporal warping algorithm, serving as a correction factor for alignment quality in subsequent cross-modal attention calculations.

[0132] Example 3: This example describes the implementation of the hierarchical contrastive learning loss function. Traditional emotion recognition models mostly use the cross-entropy loss function, which has a weak ability to constrain the structure of the feature space and easily leads to blurred boundaries between easily confused emotion categories such as "anger" and "disgust" in the feature space.

[0133] Sentence-level cohesion loss Within a training batch, for each sample, its global sentiment representation vector is used as the anchor point. Samples with the same label within the batch are considered positive examples, and samples with different labels are considered negative examples. The cosine similarity between the anchor point and the positive examples is calculated and normalized. Temperature hyperparameter Set to 0.5.

[0134] Sequence-level smoothing loss Segmentation of long speech sequences Each segment constrains the probability distribution of sentiment prediction at adjacent time steps to prevent drastic changes.

[0135] The total loss function is .

[0136] For the easily confused categories of "anger" and "disgust", the introduction of hierarchical contrastive learning loss increased the average inter-class distance between the two categories by about 41%, and reduced the confusion rate from 21.3% to 9.6%.

[0137] Example 4: This example is based on Example 1, focusing on the energy constraint weighting coefficient in step S4. The problem of insufficient adaptability in complex acoustic environments, and the modal logic conflict suppression factor. To address the robustness bottleneck caused by the independence of alignment paths, a dynamic time warping mechanism with adaptive energy constraints is proposed.

[0138] This embodiment establishes a mapping function from acoustic modal trust to physical alignment space, enabling the cross-modal alignment process to have environmental awareness.

[0139] In step S42 of Example 1, the alignment cost is defined as Among them, the energy constraint weights It is a constant value (0.15).

[0140] When the signal-to-noise ratio (SNR) of the original speech signal is below 15dB (such as street background noise or factory machinery noise), the short-time energy of the acoustic frame... It contains a large amount of non-human noise energy. At this time, if If kept constant, the dynamic time warping algorithm can be misled by noise energy, forcibly aligning high-noise frames with text keywords, causing the alignment path to shift to the noise energy peak rather than the true semantic stress. Conversely, in a clean recording studio environment with a high signal-to-noise ratio (>25dB), a fixed... It is impossible to fully utilize high-precision energy envelope information to refine alignment boundaries.

[0141] To address the aforementioned issues, this embodiment uses the acoustic modal confidence level calculated in step S45. Feedforward is applied during the alignment phase to construct adaptive energy-constrained weights. Its core logic is: when acoustic credibility is low, weaken the drag force of the energy term on the dynamic time warping path, and force the model to revert to alignment dominated by feature semantic similarity.

[0142] The specific steps are as follows: Step S421: Calculate the frame-level adaptive constraint factor. Introduce a Sigmoid-type mapping function to adjust the acoustic modal confidence. Mapped to the overall constrained tone of the current speech segment And fine-tuning is performed based on the local signal-to-noise ratio fluctuations of the frame. Adaptive energy-constrained weights. The calculation formula is:

[0143] in: The maximum constraint limit is set to 0.3 in this embodiment to prevent the energy term from excessively dominating the alignment under any circumstances; The steepness factor is set to 10, so that the acoustic modal confidence level is within the threshold. A noticeable switching effect is generated nearby; The trust threshold is set to 0.5. When When the first factor approaches 0, it significantly reduces the impact of energy constraint; For the first The instantaneous signal-to-noise ratio estimate of the frame is calculated in real time using an improved minimum statistical noise estimation method; For reference signal-to-noise ratio, it is set to the standard speech signal-to-noise ratio of 25dB; To prevent extremely small constants from being divided by zero, it is set to... .

[0144] Step S422: Construct the alignment cost matrix for enhanced physical perception. The original fixed... Replace with Corrected alignment cost for:

[0145] in, This is not the original noisy frame energy, but rather an effective speech energy estimate after spectral subtraction preprocessing. This process removes the energy basis of steady-state noise, making the absolute value of the energy difference more accurate. It truly reflects the physical correspondence between the intensity of pronunciation and the semantic focus, rather than the random disturbance of environmental noise.

[0146] In Example 1, the alignment path The determination of the alignment confidence level is decoupled from the subsequent calculation of the conflict suppression factor. In this embodiment, the cumulative distance on the alignment path is normalized to the alignment confidence level, and this confidence level is used as prior knowledge to intervene in the generation logic of the conflict suppression factor.

[0147] Step S423: Calculate the prior confidence of the alignment path To obtain the optimal alignment path Then, calculate the average normalized cumulative distance of the path. Because low-quality speech (such as heavy accents or extremely fast speech) still has a significantly higher cumulative path distance than standard speech, even after dynamic time-warping alignment. (This is followed by a definition of alignment reliability.) for:

[0148] in To align the upper limit of tolerance, this embodiment is set to 2.0. The higher the value, the stronger the intrinsic consistency between acoustic and textual modalities in the feature space.

[0149] Step S424: Integrate conflict suppression enhancement logic with prior confidence. Modify the original conflict suppression factor. The computational logic. When modal logic inconsistency is detected. Exceeding the threshold At that time, no longer simply based on Instead of determining the intensity of suppression based on text trust, alignment trust is introduced. A second arbitration is conducted. Enhanced logical conflict suppression factor. The calculation formula is:

[0150] Logical function analysis: If the alignment path is highly reliable ( However, there is a serious conflict between the two modalities in emotional tendencies. ),but ,at this time Although the content is ironic, the speaker clearly performs the physical actions (emphasis, prolongation) to express the irony. At this point, the system should trust this deliberate acoustic modification, not completely suppress the acoustic modality, but retain this "insincere" acoustic evidence for subsequent classifiers to make a comprehensive judgment.

[0151] If the alignment path has low reliability ( Furthermore, if the modalities tend to conflict, then... , The decrease is significant. This indicates that the so-called "conflicts" are most likely pseudo-conflicts caused by alignment misalignment or recognition errors due to environmental noise. In this case, acoustic modes should be strongly suppressed to prevent erroneous sentiment features caused by noise from contaminating the fusion results.

[0152] In practical implementation, this solution introduces two specific functional modules at the system level to further clarify the protection boundaries: Adaptive energy constraint control unit: Located within the cross-modal nonlinear alignment and fusion module. This unit receives the acoustic modal confidence level output by the modal confidence evaluation unit. Through mapping function The energy constraint weight coefficients acting on the dynamic time warping cost matrix are dynamically calculated to achieve the technical effect of automatically weakening physical energy alignment constraints in low signal-to-noise ratio scenarios and enhancing semantic accent alignment accuracy in high signal-to-noise ratio scenarios.

[0153] Alignment quality feedforward arbitration submodule: Located between the logical conflict detection unit and the adaptive weight calculation unit. This submodule is used to extract the normalized cumulative distance of the optimal alignment path and convert it into alignment confidence. Based on the alignment credibility, the logical conflict suppression factor is weighted and modified to distinguish between pseudo-conflicts caused by environmental interference and true conflicts caused by ironic expressions, thereby improving the robustness of the conflict correction mechanism in non-ideal acoustic environments.

[0154] To verify the effectiveness of this embodiment, under the same experimental settings (IEMOCAP and real call center noise dataset), a special test on "low signal-to-noise ratio ironic scenario" was added (simulating ironic statements under 70dB background noise on the street).

[0155] Test results show that: compared to Example 1, the fixed The strategy in this embodiment improves alignment accuracy by approximately 6.2% (from 87.4% to 93.6%). In the "low signal-to-noise ratio irony scenario," the F1 score for emotion recognition in Embodiment 4 is 11.5% higher than that in Embodiment 1. This demonstrates that introducing dynamic energy constraints and prior confidence arbitration can effectively solve the problems of physical alignment failure and logical misjudgment in noisy environments.

[0156] Example 5: Based on Example 4, this example provides a solution based on the micro-prosodic tension index to address the technical problem of the failure of pure physical energy constraints in the scenario of inhibitory emotional expression in step S4.

[0157] In real-world applications, negative emotions such as anger and sadness are often expressed through restraint and suppression, a phenomenon known as "cold anger." Speakers consciously limit airflow from their lungs to lower their volume, causing short-term energy characteristics to flatten out over time and remain at a low level for an extended period. In this situation, dynamic time warping algorithms that rely on energy peaks to anchor semantic focus lose their physical traction. The model is highly susceptible to interference from occasional high-energy pulses in the environmental background, leading to misalignment of the nonlinear mapping path. The sound pressure level, measured purely in a physical dimension, can no longer fully reflect the biomechanical tension of the human vocal organs under extreme emotional suppression.

[0158] To reconstruct alignment anchor points within the low-energy range, this embodiment mines the microscopic non-stationary fluctuations of the vocal cords' motion, using them as the second-dimensional physical constraint of the alignment cost matrix. The system extracts the fundamental frequency perturbation sequence of acoustic features for each frame in real time. With spectral flux sequence The fundamental frequency perturbation sequence is used to quantify the minute, irregular vibrations of the vocal cord vibration period on the time scale, with an extraction time window set to 40 ms. The spectral flux sequence is obtained by calculating the Euclidean distance of the short-time Fourier transform amplitude spectrum between two adjacent frames, used to measure the intensity of high-frequency harmonic energy transfer. Due to excessive tension in the vocal cord muscles when suppressing emotions, even at very low volumes, high-frequency breaks still occur in the high-frequency overtone region, which is reflected in the data distribution as follows: and The price rose in tandem.

[0159] The system extracts the micro-prosodic tension index through a nonlinear mapping function composed of two fully connected neural networks. The calculation formula is defined as follows: .in It takes values ​​between 0 and 1 and is dimensionless. The weight matrix is ​​dimensionless; This indicates that the standardized perturbation sequence and the flux sequence are concatenated along the feature dimension. Scalar bias, dimensionless; This represents the Sigmoid activation function.

[0160] A logical triggering mechanism is introduced to avoid computational redundancy caused by micro-features in regular speech segments. The system calculates the global effective energy distribution of the current speech segment and sets an active energy threshold. Set to the 15th percentile of the global energy sort. Tension trigger threshold. Set to 0.65. During the alignment process, a dual condition check is performed when scanning each frame: the frame's short-time energy is checked if and only if... And its micro-rhythmic tension index When the time step is determined to be a hidden emotional outburst point of "low sound pressure - high tension", the system activates the energy constraint compensation mechanism.

[0161] Matrix nodes activated by this mechanism The system reconfiguration alignment cost calculation formula is as follows:

[0162] in To correct for alignment costs, it is dimensionless; and Maintain the original feature mapping output; It is a natural exponential function; The tension amplification factor is set to 2.0 in this embodiment based on the cross-validation results; it is dimensionless.

[0163] Exponential amplification term The energy difference base at the bottom is amplified geometrically. A frame of speech with only 20% of the normal volume, driven by a tension exponent as high as 0.8, exerts an effect in the cost matrix equivalent to a frame of extreme plosive. This constraint forces the minimum cumulative path of dynamic time warping to traverse these "quiet but tense" time nodes, achieving precise anchoring of textual semantics and genuine emotional vocalization, cutting off the possibility of environmental noise taking over the alignment path.

[0164] Example 6: This example discloses a local sentiment gradient superposition mechanism in the modal logic inconsistency calculation process, which aims to handle the fine-grained conflict omissions caused by the global divergence calculation of the original scheme.

[0165] In long-sequence interactions, insincere emotional expressions often exhibit strong local characteristics. For example, a user may state a long sentence in a calm and objective tone, only showing a sudden reversal and exaggeration in the tone of the last word at the end of the sentence.

[0166] Example 1 uses bidirectional Kohlbek-Leibler divergence to integrate the probability distribution of the entire sentence, calculating the average divergence over the entire time axis. Acoustic abrupt changes, whose local duration accounts for less than 10% of the total sentence length, are easily diluted in the probability space by a stable, neutral preceding state lasting for tens of frames. The global divergence value ultimately remains within a safe range, causing the logical conflict suppression factor to fail to trigger, and the enhanced sentiment feature matrix still absorbs the masked contradictory features.

[0167] The system extracts multi-scale discrete difference variables along the time step sequence. Acoustic local emotion evolution gradient vector. Defined as time With time The difference vector of acoustic sentiment probability distribution. Similarly, the local sentiment evolution gradient vector of the text is obtained. This set of discrete difference operations strips away the static absolute value of probability, directly obtaining the tangent slope vectors of different modalities on the trajectory of sentiment tendency evolution.

[0168] The system calculates the angle mapping of different modal evolution gradients in geometric space, i.e., the local dynamic conflict penalty sequence. :

[0169] in It is a cosine distance metric, dimensionless; the numerator performs the inner product of two gradient vectors; the denominator is the product of the L2 norms of the two gradient vectors. It is a dimensionless mathematical smoothing factor used to prevent numerical overflow caused by the denominator approaching zero. If the emotional trends of the acoustic modality and the text modality are consistent, their gradient vectors are collinear, and the cosine distance approaches 0; if the text is stable (with a very small gradient) while the acoustic modality is abrupt (with a huge gradient), or if their evolution directions are opposite, the cosine distance approaches or even exceeds 1.

[0170] Configure a sliding time window technique to capture local extrema over a global span. The system sets a sliding window with a length of 50 frames (equivalent to a 0.5-second time span). The step size is set to 10 frames. Calculate the maximum peak value of the sequence. At this point, no matter how short the physical time corresponding to the extreme value is, it can be captured without loss during the scanning of the sliding window.

[0171] The model transforms this into a dynamic divergence gain term and reorganizes the global conflict computation architecture. The updated modal logical inconsistency... for:

[0172] in It takes values ​​between 0 and 1 and is dimensionless. The evolutionary penalty coefficient is set to 1.5 in this embodiment, and is dimensionless. Once a high-frequency local mutation is captured, The term, through gain amplification, directly drives the constant within the parentheses and pushes it past the high-level threshold. After compression of the logistic regression function, Forced to rise to a preset threshold Above (i.e., 0.6), the generation logic of the logic conflict suppression factor is directly activated, and the modal interference with abnormal mutations is adaptively stripped away.

[0173] Example 7: In this example, at the data organization level, based on the original 15,000 training corpora, 1,800 test subsets with "low sound pressure" and "ironic ending" attributes were screened and cleaned from the public database IEMOCAP and real intelligent cockpit sequences. Among them, the short-time energy of the "suppressed anger" sample group, after manual verification, was in the bottom 20% range of the global sequence distribution; the "local sudden irony" sample group showed a stable emotional semantics at the beginning and end of the text, but with highly abrupt changes in the acoustic spectrum.

[0174] Model training utilizes dual NVIDIA A100 TensorCore GPUs for parallel data processing. The batch size per GPU is configured to 64. The Adaptive Momentum Estimation (AdamW) algorithm is used for backpropagation of the joint loss function. The initial learning rate is set to... A cosine annealing learning rate decay strategy is introduced to prevent parameter oscillations near local minima. The network converges by triggering an early stopping mechanism at the 120th iteration.

[0175] Table 2 presents the comparative data of model testing using the control variable method for the newly added mechanism:

[0176] The basic control group is the state of Example 1 in the original application. When faced with specific boundary data, its recognition accuracy for low sound pressure levels drops to 74.2%. After introducing the micro-prosodic tension index of Example 5, the low sound pressure emotion recognition accuracy of mechanism stripping group A jumps to 86.7%, confirming that the reconstruction term based on high-frequency information of vocal cord vibration effectively reshapes the nonlinear alignment path of weak energy segments. After introducing the mechanism of Example 6, mechanism stripping group B records a recognition accuracy of 88.4% in the local burst irony test domain, confirming that the cosine distance gradient capture algorithm can capture tiny emotional mutation points submerged in long sequences without loss, and thus make a correct judgment on modal inconsistency. The complete combination group runs two sets of constraint algorithms simultaneously, and the comprehensive F1 score under complex speech morphology interaction breaks through to 90.3%.

[0177] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0178] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. It should be noted that any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A deep learning-based method for sentiment analysis of speech content, characterized in that, Includes the following steps: Step S1: Obtain the original speech signal to be analyzed, preprocess the original speech signal to construct an acoustic feature sequence, and simultaneously use a speech recognition engine to convert the original speech signal into a text sequence. Step S2: Construct a deep semantic enhancement model, input the text sequence into the deep semantic enhancement model for multi-level feature mapping, and extract deep text semantic feature vectors containing contextual logical relationships; Step S3: Construct a multi-scale acoustic spatiotemporal feature extraction model. Through a parallel temporal convolutional network and a recurrent neural network with a self-attention mechanism, perform joint feature extraction in the time and frequency domains of the acoustic feature sequence to generate an acoustic physical representation feature vector. Step S4: Establish a cross-modal nonlinear alignment and logical conflict correction mechanism: Using a dynamic time warping algorithm based on energy distribution constraints, find the optimal nonlinear mapping path between the acoustic physical representation feature vector and the deep text semantic feature vector to achieve precise alignment between the acoustic modal sequence and the text modal sequence; calculate the relative entropy between the sentiment distribution of the acoustic modal sequence and the sentiment distribution of the text modal sequence to obtain the modal logical inconsistency; when the modal logical inconsistency exceeds a preset threshold, generate a logical conflict suppression factor to correct the fusion weights; simultaneously, calculate the acoustic modal confidence based on the signal-to-noise ratio of the original speech signal and the energy distribution of the acoustic feature sequence, calculate the text modal confidence based on the word-level confidence output by the speech recognition engine, and dynamically calculate adaptive fusion weights based on the acoustic modal confidence, the text modal confidence, and the logical conflict suppression factor to generate an enhanced sentiment feature matrix; Step S5: Construct a long-sequence emotion evolution modeling layer based on global tone anchoring. Input the enhanced emotion feature matrix into the combined architecture of bidirectional long short-term memory network and global context memory unit. Extract the global emotion tone vector of the sequence through global average pooling. Adaptively fuse the global emotion tone vector with the instantaneous hidden state of the bidirectional long short-term memory network using a gating mechanism to capture the dynamic fluctuation trajectory of the emotion state over time and output the global emotion representation vector. Step S6: Perform nonlinear classification mapping on the global sentiment representation vector using an end-to-end sentiment classifier to output the final sentiment analysis result.

2. The deep learning-based speech content sentiment analysis method according to claim 1, characterized in that, In step S1, the preprocessing step of the original speech signal specifically includes: Step S11: Perform pre-weighting processing on the mono original speech signal with a sampling frequency of 16kHz and a sampling bit depth of 16bit, and set the pre-weighting coefficient to 0.

97. Step S12: The pre-weighted signal is framed using a Hamming window, with a frame length of 25ms and a frame shift of 10ms, ensuring an inter-frame overlap rate of 50%. Step S13: Perform a fast Fourier transform on each frame of signal, and map the transformed signal to a Mel scale filter bank to extract 40-dimensional Mel frequency cepstral coefficients and their first-order and second-order difference features, forming a 120-dimensional acoustic feature sequence.

3. The deep learning-based speech content sentiment analysis method according to claim 1, characterized in that, In step S2, the deep semantic enhancement model adopts a 12-layer bidirectional transformation encoder structure, with the hidden layer dimension set to 768 and the number of heads for multi-head attention set to 12. The deep semantic enhancement model internally captures long-distance semantic dependencies in the text content through a multi-head self-attention mechanism.

4. The deep learning-based speech content sentiment analysis method according to claim 1, characterized in that, In step S3, the operation flow of the multi-scale acoustic spatiotemporal feature extraction model includes: Step S31: Input the acoustic feature sequence into a temporal convolutional network consisting of four layers of dilated convolutions, with the dilation factor of each layer increasing in a power of two order. Step S32: Simultaneously input the acoustic feature sequence into the recurrent neural network, and use the gated recurrent unit to perform temporal modeling of the pitch fluctuations of the speech signal; Step S33: The output of the temporal convolutional network and the output of the recurrent neural network are aggregated through a weighted fusion layer to generate the acoustic physical representation feature vector with consistent dimensions.

5. The deep learning-based speech content sentiment analysis method according to claim 1, characterized in that, In step S4, the specific method for finding the optimal nonlinear mapping path using the dynamic time warping algorithm based on energy distribution constraints is as follows: The alignment cost is calculated as follows: the square of the Euclidean distance between the acoustic feature vector after feature mapping and the word embedding vector after mapping is added to the product of the energy constraint weight coefficient and the absolute value of the energy difference; wherein, the absolute value of the energy difference is the absolute value of the difference between the short-time energy of the acoustic feature of the frame and the average energy reference value of the time interval corresponding to the word.

6. The deep learning-based speech content sentiment analysis method according to claim 5, characterized in that, The optimal nonlinear mapping path is found recursively by finding the path with the minimum cumulative distance. The specific recursive logic is as follows: the cumulative distance of a grid point is equal to the alignment cost of the current grid point, plus the minimum cumulative distance among its three adjacent grid points to the left, below, and to the lower left. After the calculation is completed, the optimal alignment path that minimizes the total cumulative distance is obtained by backtracking.

7. The deep learning-based speech content sentiment analysis method according to claim 1, characterized in that, In step S4, the modal logic inconsistency degree is calculated as follows: The aligned acoustic modal sequence and text modal sequence are input into the sentiment probing head, and the acoustic sentiment probability distribution and text sentiment probability distribution are output. Calculate the sum of the two-way Kohlbek-Leibler divergences between the two probability distributions mentioned above, and map this sum to the interval between zero and one using a logistic regression function, which serves as the modal logical inconsistency degree.

8. The deep learning-based speech content sentiment analysis method according to claim 7, characterized in that, In step S4, the generation logic of the logical conflict suppression factor is as follows: When the modal logic inconsistency exceeds a preset threshold, the logic conflict suppression factor is equal to the product of the conflict sensitivity coefficient and the inconsistency exceeding the threshold. When the modal logic inconsistency does not exceed a preset threshold, the logic conflict suppression factor is equal to one.

9. The deep learning-based speech content sentiment analysis method according to claim 1, characterized in that, In step S4, the step of finding the optimal nonlinear mapping path using a dynamic time warping algorithm based on energy distribution constraints further includes: Acquiring acoustic modal trust and frame-level instantaneous signal-to-noise ratio The energy constraint weights acting on the alignment cost matrix are dynamically calculated using an adaptive mapping function. The adaptive mapping function makes the energy constraint weights approach zero when the acoustic modal confidence is lower than a preset threshold, and enhances the alignment constraint contribution of the absolute value of the energy difference term in a high signal-to-noise ratio environment. The alignment cost calculation formula is revised as follows: ,in This is the effective speech energy estimation after preprocessing by spectral subtraction.

10. A deep learning-based speech content sentiment analysis system, used to execute the deep learning-based speech content sentiment analysis method according to any one of claims 1-9, characterized in that, include: The data acquisition and preprocessing module is used to acquire the raw speech signal and perform preprocessing, extract acoustic feature sequences, and generate the corresponding text sequences; The semantic enhancement extraction module is used to perform deep semantic mapping on the text sequence and output a deep text semantic feature vector. A multi-scale acoustic feature module is used to extract acoustic physical representation feature vectors from the acoustic feature sequence; The cross-modal nonlinear alignment and fusion module is used to establish a nonlinear alignment mechanism based on energy distribution constraints, and combines the inhibition factor generated by modal trust assessment and logical conflict detection to dynamically calculate the fusion weight and output the enhanced sentiment feature matrix. The global tone anchoring evolution module is used to extract the global sentiment tone vector using global average pooling and to fuse it with the hidden state through a gating mechanism to capture the dynamic evolution of sentiment. The Emotion Decision Classification module is used to map the global emotion representation vector to a specific emotion category and output the result.

Images