A customer service dialogue sentiment analysis method and system based on multi-modal time sequence perception

Abstract

This invention provides a method and system for sentiment analysis in customer service dialogues based on multimodal temporal awareness. The method involves acquiring physical tremor features in customer service speech, focusing on these features to generate denoised physical tremor features, performing feature transformation on the denoised physical tremor features to generate sentiment feature vectors, quantizing the temporal dialogue text during the customer service conversation to generate text feature vectors, aligning the sentiment feature vectors and text feature vectors with timestamps to generate multimodal temporal fusion features, and performing state analysis on the multimodal temporal fusion features to generate a sentiment analysis scheme for conducting sentiment analysis in customer service dialogues. This invention overcomes three major shortcomings in customer service scenarios: speech tremor interference, text semantic lag, and cross-modal temporal misalignment. It achieves second-level accurate tracking of emotional fluctuations during the dialogue, providing real-time data support for service quality assessment.

Description

Technical Field

[0001] This invention relates to the field of customer service dialogue sentiment analysis technology, and in particular to a customer service dialogue sentiment analysis method and system based on multimodal temporal awareness. Background Technology

[0002] With the widespread application of intelligent customer service systems, customer service personnel are working under high-intensity call loads, leading to increasingly prominent issues of emotional exhaustion. This emotional state not only affects the quality of communication but may also result in decreased customer satisfaction. Therefore, there is an urgent need in practical applications for a technological solution that can accurately analyze emotional changes during customer service conversations, thereby providing data support for subsequent intervention and management.

[0003] To address the aforementioned technical needs, deep learning-based single-modal speech emotion recognition solutions have been applied to customer service emotion monitoring. These solutions extract acoustic features from customer service speech signals and use pre-trained neural network models to classify emotional states, thereby achieving dynamic tracking of customer service emotions and, to some extent, automated emotion recognition. However, existing solutions have some inherent limitations. These include relying solely on speech signals for emotion judgment, ignoring the emotional information inherent in the customer service representative's language during the conversation. This leads to misjudgments or missed judgments when faced with a calm tone but negative semantics or complex emotions. Furthermore, the lack of effective modeling of the time-series context makes it difficult to capture the trend characteristics of emotional evolution, limiting their stability and accuracy in long-duration, multi-turn dialogue scenarios. Summary of the Invention

[0004] This invention provides a customer service dialogue sentiment analysis method and system based on multimodal temporal awareness, which solves the problems of existing technologies that rely solely on voice signals for emotion judgment, ignoring the emotional information contained in the language content of customer service during the dialogue, leading to misjudgment or omission when facing situations with a calm tone but negative semantics or complex emotions; lacking effective modeling of time series context, it is difficult to capture the trend characteristics of emotion evolution, which limits its stability and accuracy in long-term and multi-round dialogue scenarios.

[0005] In a first aspect, the present invention provides a customer service dialogue sentiment analysis method based on multimodal temporal awareness, comprising:

[0006] The physical vibration features in the customer service voice are obtained by using a preset acoustic sensor array, and the physical vibration features are focused to generate noise-reduced voice physical vibration features.

[0007] Based on a preset real-time emotional state classification circuit, the physical vibration features of the denoised speech are transformed to generate an emotional feature vector.

[0008] The time-series dialogue text during customer service conversations is obtained, and the time-series dialogue text is quantized to generate a text feature vector.

[0009] The emotion feature vector and the text feature vector are timestamped to generate multimodal temporal fusion features;

[0010] The multimodal temporal fusion features are analyzed and processed using a preset multimodal temporal perception model to generate a sentiment analysis scheme for conducting sentiment analysis in customer service dialogues.

[0011] Optionally, physical flutter features in customer service voice are acquired through a preset acoustic sensor array, and the physical flutter features are focused to generate noise-reduced voice physical flutter features, including:

[0012] Customer service voice signals are collected by sensor units in a preset acoustic sensor array, and oscillation components with an oscillation frequency range of 50Hz to 150Hz are extracted from the unit signals of each sensor unit to generate the initial physical vibration characteristics of each sensor unit.

[0013] Calculate the phase difference between each of the initial physical tremor features, and locate the spatial source direction of the customer service voice signal based on the phase difference;

[0014] Based on the spatial source direction, oscillation component weights are assigned to each of the sensor units to generate weighted oscillation components;

[0015] The weighted oscillation components are aggregated to generate an aggregated oscillation signal;

[0016] Continuous oscillation segments with amplitude fluctuation intensity greater than a preset fluctuation threshold are selected from the aggregated oscillation signal, and these continuous oscillation segments are marked as speech physical tremor features after noise reduction.

[0017] Optionally, based on a preset real-time emotion state classification circuit, the denoised speech physical tremor features are transformed to generate an emotion feature vector, including:

[0018] Based on a preset real-time emotional state classification circuit, the denoised speech physical tremor features are divided into multiple continuous feature segments of equal time length.

[0019] Calculate the maximum and minimum amplitude values ​​of each continuous feature segment, and perform a difference operation on the maximum and minimum amplitude values ​​to generate the amplitude fluctuation intensity of the continuous feature segment;

[0020] Based on the time sequence of the amplitude fluctuation intensity, three consecutive amplitude fluctuation intensities are selected, and the slope of the change of the three consecutive amplitude fluctuation intensities is calculated to generate a dynamic change trend.

[0021] Based on the dynamic change trend, weighting factors are assigned to the three consecutive feature segments corresponding to the three consecutive amplitude fluctuation intensities to generate weighted feature segments;

[0022] Each of the weighted feature segments is connected to generate an emotion feature vector.

[0023] Optionally, the time-series dialogue text during the customer service conversation is obtained, and the time-series dialogue text is quantized to generate a text feature vector, including:

[0024] The customer service dialogue text is segmented according to a preset time window to generate multiple text units arranged in chronological order.

[0025] Extract a set of words containing sentiment from each of the text units to generate a set of keywords for multiple text units;

[0026] The frequency of occurrence of each keyword in each keyword set is multiplied by a preset sentiment weight factor, and the results of each multiplication are added together to generate a numerical quantification result of multiple text units.

[0027] Each of the numerical quantization results is combined based on time order to generate a text feature vector.

[0028] Optionally, the emotion feature vector and the text feature vector are timestamped to generate multimodal temporal fusion features, including:

[0029] Extract the timestamp corresponding to each emotion feature point in the emotion feature vector to generate an emotion timestamp sequence;

[0030] Extract the timestamp corresponding to each text feature point in the text feature vector to generate a text timestamp sequence;

[0031] The overlapping time points of the emotion timestamp sequence and the text timestamp sequence are matched to generate matching timestamp pairs;

[0032] Based on the matching timestamp pair, synchronous feature extraction operations are performed on the emotion feature vector and the text feature vector respectively to generate aligned emotion feature points and aligned text feature points respectively;

[0033] The aligned emotion feature points and the aligned text feature points are concatenated to generate multimodal temporal fusion features.

[0034] Optionally, matching the overlapping time points of the emotion timestamp sequence and the text timestamp sequence to generate matching timestamp pairs includes:

[0035] Calculate the absolute time difference between each emotion timestamp in the emotion timestamp sequence and each text timestamp in the text timestamp sequence to generate an absolute time difference set;

[0036] From the set of absolute time differences, target absolute time differences that are less than a preset time difference tolerance range are selected to generate a candidate time difference set;

[0037] The average value of the emotion timestamp and text timestamp corresponding to each time difference in the candidate time difference set is calculated to generate a matching timestamp pair.

[0038] Optionally, the multimodal temporal fusion features are processed by a preset multimodal temporal perception model to generate a sentiment analysis scheme for customer service dialogue sentiment analysis, including:

[0039] The multimodal temporal fusion features are segmented into analysis units of fixed time length based on time order, and the amplitude-weighted average of all feature points in each analysis unit is calculated to generate unit fusion features.

[0040] The unit fusion feature of the current analysis unit is linearly superimposed with the state memory feature of the previous analysis unit to generate the state memory feature of the current analysis unit;

[0041] The state memory features are compared with a preset set of emotional benchmark features to generate an emotional tendency probability distribution.

[0042] A trend consistency detection is performed on the probability distribution of sentiment tendency in three consecutive analysis units to generate sentiment state transition markers;

[0043] Each emotional state transition marker is combined based on time sequence to generate a sentiment analysis scheme for conducting sentiment analysis of customer service conversations.

[0044] Secondly, the present invention provides a customer service dialogue sentiment analysis system based on multimodal temporal awareness, comprising:

[0045] The focusing module is used to acquire physical vibration features in customer service voice through a preset acoustic sensor array, and to focus the physical vibration features to generate noise-reduced voice physical vibration features.

[0046] The conversion module is used to perform feature conversion on the physical tremor features of the denoised speech based on a preset real-time emotional state classification circuit, and generate an emotional feature vector.

[0047] The quantization module is used to acquire the time-series dialogue text during customer service conversations, perform quantization processing on the time-series dialogue text, and generate a text feature vector.

[0048] The alignment module is used to perform timestamp alignment processing on the emotion feature vector and the text feature vector to generate multimodal temporal fusion features;

[0049] The analysis module is used to perform state analysis processing on the multimodal temporal fusion features through a preset multimodal temporal perception model, and generate a sentiment analysis scheme for conducting sentiment analysis of customer service dialogues.

[0050] Thirdly, the present invention provides a computing device including a processor and a memory, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute a customer service dialogue sentiment analysis method based on multimodal temporal awareness as described in any of the first aspects.

[0051] Fourthly, the present invention provides a computer storage medium storing computer program instructions thereon, wherein the computer program instructions, when executed by a processor, implement the customer service dialogue sentiment analysis method based on multimodal temporal awareness as described in any one of the first aspects.

[0052] This invention captures and focuses on denoising the physical vibration features of speech through an acoustic sensor array, combines this with a real-time emotional state classification circuit to convert emotional features, synchronously quantifies the temporal dialogue text to generate text features, and then fuses multimodal temporal features through timestamp alignment. Finally, a multimodal temporal perception model dynamically analyzes the emotional state, directly overcoming the three major defects in customer service scenarios: speech vibration interference, text semantic lag, and cross-modal temporal misalignment. It achieves second-level accurate tracking of emotional fluctuations during dialogue, providing real-time data support for service quality assessment.

[0053] Furthermore, the initial physical tremor features are generated by extracting oscillation components between 50Hz and 150Hz through a sensor array. The direction of the sound source is located by using phase difference and component weights are assigned. Signal aggregation is performed on the weighted oscillation components to enhance the target voiceprint. Then, continuous oscillation segments with amplitudes exceeding the threshold are screened to mark noise reduction features. This completely eliminates the pollution of voice tremor features by background noise in the customer service environment, ensures the physical purity of the input features of the emotion classification circuit, and significantly reduces the error rate of voice emotion analysis.

[0054] These or other aspects of the invention will become more apparent from the following description of the embodiments. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 A flowchart illustrating a customer service dialogue sentiment analysis method based on multimodal temporal awareness, provided as an embodiment of the present invention;

[0057] Figure 2 A schematic diagram of the structure of a customer service dialogue sentiment analysis system based on multimodal temporal awareness provided in an embodiment of the present invention;

[0058] Figure 3 This is a schematic diagram of the structure of a computing device provided in an embodiment of the present invention. Detailed Implementation

[0059] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0060] In some of the processes described in the specification, claims, and accompanying drawings of this invention, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] Figure 1 A flowchart of a customer service dialogue sentiment analysis method based on multimodal temporal awareness is provided as an embodiment of the present invention, such as... Figure 1 As shown, the method includes:

[0063] To address the challenge of detecting emotional exhaustion caused by prolonged, high-load calls in customer service scenarios, existing technologies suffer from three main shortcomings: First, voice emotion analysis relies on traditional voiceprint features, failing to capture low-frequency physical tremors caused by fatigue; second, text semantic analysis lags behind changes in voice emotion, leading to cross-modal feature misalignment; and third, single-modal detection ignores the temporal correlation between voice tremors and dialogue content, mistakenly identifying physiological vocal cord tremors as emotional fluctuations. These shortcomings result in a false negative rate of up to 35% for emotional exhaustion states, failing to meet the real-time early warning needs of call centers. To address these issues, this invention proposes the following approach: It uses an acoustic sensor array to directionally capture the unique physical oscillation characteristics of customer service representatives' vocal cord tremors, followed by spatial focusing to remove environmental noise; a dedicated circuit converts the noise-reduced tremor features into an emotion vector, simultaneously extracting the temporal quantization features of the dialogue text; a timestamp-level alignment mechanism for dual-modal features is established to eliminate response delays between voice and text; finally, a multimodal temporal perception model dynamically tracks the coupling evolution of tremor intensity and dialogue content, accurately identifying the continuous state transition from occupational fatigue to emotional exhaustion, providing a minute-level response window for intervention in overloaded work. Based on this, the present invention provides a customer service dialogue sentiment analysis method based on multimodal temporal awareness, such as... Figure 1 ,include:

[0064] Step 101: Obtain physical vibration features in customer service voice through a preset acoustic sensor array, and focus on the physical vibration features to generate noise-reduced voice physical vibration features.

[0065] In this step, the preset acoustic sensor array refers to a microphone group distributed according to spatial geometric rules, including at least three sensor units, used to directionally acquire speech signals and suppress environmental noise; physical tremor characteristics refer to the 50 to 150 Hz low-frequency oscillation components generated by vocal cord vibration, reflecting the physiological emotional fluctuation characteristics of customer service personnel; focusing operation refers to adjusting sensor weights and focusing signals based on the direction of the sound source, used to enhance the intensity of target speech tremor; the noise-reduced speech physical tremor characteristics refer to the pure oscillation segments after spatial focusing and amplitude filtering, containing effective tremor information related to emotions.

[0066] In this embodiment of the invention, customer service voice signals are firstly acquired synchronously by multiple spatially distributed sensor units in a preset acoustic sensor array. Secondly, specific low-frequency oscillation components are extracted from the signals of each sensor unit as initial physical flutter features. Then, the phase difference between all initial physical flutter features is calculated and the spatial direction of the sound source is located. Next, weights are assigned to the oscillation components of each sensor unit based on this direction and a signal aggregation operation is performed to generate aggregated oscillation signals. Finally, oscillation segments in the aggregated oscillation signals whose amplitude continuously exceeds a preset threshold are selected and marked as denoised voice physical flutter features.

[0067] Step 102: Based on the preset real-time emotional state classification circuit, the physical tremor features of the denoised speech are transformed to generate an emotional feature vector.

[0068] In this step, the preset real-time emotional state classification circuit refers to a hardware acceleration processor, including a feature segmentation module and a weight allocation module, which is used to convert tremor features in real time; the feature conversion operation refers to the process of generating weighted feature segments by calculating the amplitude fluctuation intensity and dynamic trend, which is used to extract temporal emotional patterns; the emotional feature vector refers to a fixed-dimensional vector formed by connecting the weighted feature segments, which encodes the temporal emotional information of voice tremor.

[0069] In this embodiment of the invention, the physical tremor features of the denoised speech are first divided into continuous feature segments with equal time spans. Then, the difference between the maximum and minimum amplitude values ​​in each feature segment is calculated to generate amplitude fluctuation intensity. Subsequently, three consecutive amplitude fluctuation intensities are selected in chronological order to calculate the slope of change and generate a dynamic trend. Then, weight factors are assigned to the corresponding feature segments based on the dynamic trend to generate weighted feature segments. Finally, the time series data of all weighted feature segments are connected to generate an emotion feature vector.

[0070] Step 103: Obtain the time-series dialogue text during the customer service conversation, quantize the time-series dialogue text, and generate a text feature vector.

[0071] In this step, quantization refers to extracting emotional words from text units and performing frequency-weighted summation to generate a numerical emotional intensity index; the text feature vector refers to a vector composed of the quantization results of text units in chronological order, representing the temporal emotional changes in the dialogue content.

[0072] In this embodiment of the invention, the text units generated in the temporal dialogue text are first divided according to a preset time window. Then, words containing sentiment tendencies in each text unit are extracted to form a keyword set. Subsequently, the sum of the product of the frequency of occurrence of each word in the keyword set and a preset sentiment weight factor is calculated to generate a numerical quantization result. Finally, the numerical quantization results of all text units are combined in chronological order to generate a text feature vector.

[0073] Step 104: Perform timestamp alignment on the emotion feature vector and the text feature vector to generate multimodal temporal fusion features.

[0074] In this step, timestamp alignment refers to generating a unified time reference by matching bimodal timestamps to eliminate cross-modal response delay; multimodal temporal fusion features refer to composite features formed by splicing emotion feature points and text feature points according to aligned timestamps, which fuse complementary information from speech and text.

[0075] In this embodiment of the invention, firstly, the timestamps of each feature point are extracted from the emotion feature vector to form an emotion timestamp sequence. Secondly, the timestamps of each feature point are extracted from the text feature vector to form a text timestamp sequence. Then, the absolute time difference of the timestamps in the two sequences is calculated and candidate pairs with a tolerance threshold are selected. Next, the arithmetic mean of the timestamps in the candidate pairs is taken to generate a matching timestamp pair. Finally, based on the matching timestamp pair, feature points in the emotion feature vector and the text feature vector are selected and concatenated to generate a multimodal temporal fusion feature.

[0076] Step 105: Perform state analysis processing on the multimodal temporal fusion features using a preset multimodal temporal perception model to generate a sentiment analysis scheme for conducting sentiment analysis of customer service dialogues.

[0077] In this step, the preset multimodal temporal perception model refers to an analysis engine that includes a state memory mechanism, used to track the coupling evolution of cross-modal features; the state analysis processing refers to generating a probability distribution by linearly superimposing historical states and current features, used to identify emotional state transition nodes; the sentiment analysis scheme refers to a time series composed of continuous emotional state transition markers, outputting the evolution path of customer emotion exhaustion.

[0078] In this embodiment of the invention, the multimodal temporal fusion features are first divided into analysis units of fixed duration. Then, the amplitude-weighted average value of feature points in each analysis unit is calculated to generate unit fusion features. Subsequently, the current unit fusion features are linearly superimposed with the state memory features of the previous unit to generate new state memory features. Next, the new state memory features are compared with a preset set of sentiment benchmark features to generate a sentiment tendency probability distribution. Then, trend consistency detection is performed on the probability distributions of three consecutive units to generate sentiment state transition markers. Finally, all markers are combined in chronological order to generate a sentiment analysis scheme.

[0079] For example, firstly, customer service representative voice recordings are collected using three spatially distributed sensors in a pre-defined acoustic sensor array. Low-frequency oscillation components are extracted from each sensor signal to generate initial physical tremor features. The direction of the sound source is located based on phase differences, and weights are assigned to generate convergent oscillation signals. Segments with amplitudes consistently exceeding a threshold are selected to output denoised tremor features. Secondly, a real-time emotion state classification circuit divides the denoised features into equal-length feature segments, calculates the amplitude fluctuation intensity and its dynamic trend, generates weighted feature segments, and connects them to form an emotion feature vector. Simultaneously, the dialogue text is segmented by second-level windows, and emotional words are extracted, frequency-weighted sums are calculated, and text feature vectors are generated. Subsequently, timestamps of bimodal feature points are extracted and subjected to tolerance matching to generate aligned timestamp pairs. Corresponding feature points are then selected and concatenated to form a multimodal fusion feature. Finally, the fusion feature is segmented into analysis units, the weighted average amplitude of each unit is calculated and superimposed with historical states, compared with a pre-defined emotion benchmark to generate a probability distribution, and continuous unit trends are detected to generate state transition markers. Combined markers are used to output an emotion exhaustion early warning scheme.

[0080] This invention employs an acoustic sensor array to directionally capture the vocal cord tremor characteristics of customer service representatives and focus on noise reduction. It combines a dedicated circuit to convert emotion vectors and text quantization features, eliminates cross-modal timing deviations through timestamp-level alignment, and uses a multimodal perception model to dynamically track the coupling evolution of tremor intensity and dialogue content. This achieves minute-level accurate early warning of emotional exhaustion under high-load call conditions, significantly reducing the missed detection rate.

[0081] To address the issue of background noise contaminating the physical flutter features of customer service speech, this step extracts oscillation components using an acoustic sensor array, assigns weights based on the direction of the sound source, aggregates the signal, and filters continuous segments to ensure the purity of the denoised features. This invention provides a specific embodiment: Step 101 involves acquiring physical flutter features in the customer service speech using a preset acoustic sensor array, focusing on these physical flutter features to generate denoised physical flutter features, specifically including the following steps:

[0082] Step 111: Acquire customer service voice signals through sensor units in a preset acoustic sensor array, and extract oscillation components with an oscillation frequency range between 50Hz and 150Hz from the unit signals of each sensor unit to generate the initial physical vibration characteristics of each sensor unit.

[0083] In this step, the oscillation component refers to the sinusoidal wave component within a specific frequency range in the speech signal, including the periodic vibration waveform separated from the original signal by a bandpass filter; the initial physical tremor feature refers to the oscillation component data extracted by a single sensor unit, which includes time amplitude sequence and phase information.

[0084] In this embodiment of the invention, customer service voice signals are first collected synchronously by sensor units in a preset acoustic sensor array. Then, a specific low-frequency oscillation component is separated from the unit signal of each sensor unit. Finally, the oscillation component is marked as the initial physical vibration feature of the corresponding sensor unit.

[0085] Step 112: Calculate the phase difference between each of the initial physical tremor features, and locate the spatial source direction of the customer service voice signal based on the phase difference.

[0086] In this step, the phase difference refers to the waveform time offset between the initial physical vibration characteristics of different sensor units, which is used to calculate the time difference of sound wave arrival; the positioning operation refers to the process of calculating the azimuth angle of the sound source based on the phase difference, including time difference measurement and direction vector calculation.

[0087] In this embodiment of the invention, the waveform phase offset between the initial physical vibration characteristics of all sensor units is first calculated as the phase difference. Then, the sound source localization calculation is performed based on the phase difference. Finally, the spatial source direction of the customer service voice signal is determined.

[0088] Step 113: Assign oscillation component weights to each sensor unit based on the spatial source direction to generate weighted oscillation components.

[0089] In this step, the oscillation component weight refers to the coefficient value assigned according to the direction of the sound source, which is used to enhance the contribution of the sensor signal in the target direction; the weighted oscillation component refers to the signal after the original oscillation component is multiplied by the weight coefficient, which reflects the spatially selective signal gain.

[0090] In this embodiment of the invention, firstly, oscillation component weighting coefficients are assigned to each sensor unit according to the spatial source direction; secondly, the original oscillation component is multiplied by the corresponding weighting coefficient; and finally, a weighted oscillation component is generated.

[0091] Step 114: Aggregate the weighted oscillation components to generate an aggregated oscillation signal.

[0092] In this step, signal aggregation refers to the processing of time-domain aligned superposition of weighted signals, including phase correction and amplitude balancing; aggregation of oscillation signals refers to the composite oscillation waveform after spatial enhancement, which fuses target features from multi-sensor data.

[0093] In this embodiment of the invention, firstly, a signal superposition operation is performed on all weighted oscillation components, secondly, the superimposed signal is normalized in amplitude, and finally, a converged oscillation signal is generated.

[0094] Step 115: Select continuous oscillation segments from the aggregated oscillation signal whose amplitude fluctuation intensity is greater than a preset fluctuation threshold, and mark the continuous oscillation segments as the physical tremor features of the speech after noise reduction.

[0095] In this step, the preset fluctuation threshold refers to the amplitude threshold value for determining valid tremors, which is set based on the statistical analysis of the intensity of vocal cord tremors of customer service personnel; the tagging operation refers to associating the selected continuous oscillation segments as noise reduction features, including timestamp binding and metadata annotation.

[0096] In this embodiment of the invention, firstly, segments in the aggregated oscillation signal whose amplitude continuously exceeds a preset fluctuation threshold are detected as continuous oscillation segments, and then these segments are marked as speech physical tremor features after noise reduction.

[0097] This invention achieves sound source directional enhancement through multi-sensor oscillation component extraction and phase difference localization, optimizes signal aggregation quality by weight allocation, retains effective vibration segments through amplitude threshold screening, and eliminates environmental interference such as keyboard tapping and air conditioner noise, thereby significantly improving the signal-to-noise ratio of customer service vocal cord vibration feature extraction.

[0098] To improve the accuracy of capturing the dynamic evolution of vocal cord tremors, this step involves dividing the vocal cords into feature segments based on an emotion classification circuit, calculating the intensity and trend of the tremors, assigning weights to generate weighted segments, and finally connecting them into an emotion vector. This invention provides a specific embodiment: Step 102, based on a preset real-time emotion state classification circuit, performs feature transformation on the denoised speech physical tremor features to generate an emotion feature vector, specifically including the following steps:

[0099] Step 201: Based on the preset real-time emotional state classification circuit, the denoised speech physical tremor features are divided into multiple continuous feature segments of equal time length.

[0100] In this step, the continuous feature segment refers to the segment generated by cutting the noise-reduced flutter feature into segments of equal duration, which includes a fixed number of sampling points and a complete amplitude waveform.

[0101] In this embodiment of the invention, the noise-reduced physical tremor features of the voice are first received through a preset real-time emotional state classification circuit. Then, the feature data is segmented into segments of fixed duration using a time-segmentation unit built into the circuit. Finally, multiple consecutive feature segments with strictly equal durations are generated.

[0102] Step 202: Calculate the maximum and minimum amplitude values ​​of each continuous feature segment, and perform a difference operation on the maximum and minimum amplitude values ​​to generate the amplitude fluctuation intensity of the continuous feature segment.

[0103] In this step, the calculation operation refers to the process of performing mathematical operations on the data, including extreme value identification, difference calculation and slope derivation; the amplitude fluctuation intensity refers to the absolute value of the amplitude range within the characteristic segment, reflecting the intensity of the energy change of vocal cord vibration.

[0104] In this embodiment of the invention, the amplitude waveform of each continuous feature segment is first scanned, then the global maximum and global minimum values ​​within the segment are identified, then the maximum and minimum values ​​are subtracted to perform a difference operation, and finally the calculation result is output as the amplitude fluctuation intensity of the continuous feature segment.

[0105] Step 203: Select three consecutive amplitude fluctuation intensities based on the time sequence of the amplitude fluctuation intensity, calculate the slope of the change of the three consecutive amplitude fluctuation intensities, and generate a dynamic change trend.

[0106] In this step, the slope of change refers to the rate of change of three consecutive fluctuation intensities, which is obtained by subtracting the previous value from the subsequent value and dividing by the time interval; the dynamic trend of change refers to the quantified value of the slope, with positive values ​​indicating an upward trend and negative values ​​indicating a downward trend, used to predict the direction of emotional evolution.

[0107] In this embodiment of the invention, three consecutive amplitude fluctuation intensity values ​​are first selected in sequence along the time axis. Then, the difference between the third value and the first value is calculated and divided by the time span. This quotient is then used as the slope of change, and finally, a dynamic trend representing the direction of fluctuation evolution is generated.

[0108] Step 204: Based on the dynamic change trend, assign weight factors to the three consecutive feature segments corresponding to the three consecutive amplitude fluctuation intensities to generate weighted feature segments.

[0109] In this step, the weighted feature segment refers to the data block after the original feature segment is multiplied by the weighting factor, which strengthens the significant sentiment segment or weakens the stable segment.

[0110] In this embodiment of the invention, a weighting factor is first determined based on the positive or negative value and magnitude of the dynamic change trend. Then, the weighting factor is multiplied by the original data of the corresponding continuous feature segment. Subsequently, an enhanced or suppressed signal segment is generated, and finally, a weighted feature segment is output.

[0111] Step 205: Connect each of the weighted feature segments to generate an emotion feature vector.

[0112] In this step, the connection operation refers to the vectorization process of splicing weighted feature segments in chronological order to maintain temporal continuity.

[0113] In this embodiment of the invention, all weighted feature segments are first arranged in chronological order, then the first and last data points of adjacent feature segments are connected, then merged into a continuous time series, and finally a uniform emotional feature vector is generated.

[0114] This invention captures local patterns of tremor features through equal-duration segmentation, quantifies changes in emotional intensity by combining amplitude fluctuation intensity, and uses dynamic trend analysis to achieve adaptive weight allocation, ultimately generating an emotional feature vector that preserves the temporal evolution pattern, thus significantly improving the detection rate of vocal cord micro-tremors caused by fatigue.

[0115] To address the challenge of implicitly recognizing expressions of occupational fatigue in text, this step involves segmenting the text by time window, extracting a set of sentiment words, weighting and summing the sentiment intensity of individual units, and then sequentially combining them into a text feature vector. This invention provides a specific embodiment: Step 103 involves acquiring the temporal dialogue text from a customer service conversation, quantizing the temporal dialogue text, and generating a text feature vector, specifically including the following steps:

[0116] Step 301: Segment the chronological dialogue text during the customer service conversation according to the preset time window to generate multiple text units arranged in chronological order.

[0117] In this step, the preset time window refers to a fixed-duration text segmentation unit, set according to the average speaking speed of customer service conversations, to ensure that each text unit contains complete semantic fragments; the chronological dialogue text refers to the text content of customer service conversations recorded in chronological order, including the interaction records of customer questions and customer service answers.

[0118] In this embodiment of the invention, the time-series dialogue text in the customer service dialogue process is first segmented according to a preset time window length, then the segmented text fragments are sorted according to the order of occurrence, and finally multiple text units are generated in strict chronological order.

[0119] Step 302: Extract a set of words containing sentiment from each text unit to generate a set of keywords for multiple text units.

[0120] In this step, the keyword set refers to the emotional vocabulary group selected from the text units, which includes emotional expression elements such as positive words, negative words, and intensity modifiers.

[0121] In this embodiment of the invention, firstly, all words in each text unit are scanned; secondly, words with emotional orientation in a preset emotional dictionary are matched; then, all successfully matched words are collected to form an independent set; and finally, this set is marked as the keyword set of the corresponding text unit.

[0122] Step 303: Multiply the frequency of each keyword in each keyword set by a preset sentiment weight factor, and add the results of each product operation to generate a numerical quantification result of multiple text units.

[0123] In this step, the preset emotional weight factor refers to the quantitative coefficient assigned to different emotional words, which is set based on the emotional intensity calibration value of the words in the customer service scenario; the numerical quantification result refers to the scalar value of the emotional intensity of the text unit, which is calculated by weighted frequency summation, reflecting the emotional tendency intensity of the dialogue during that period.

[0124] In this embodiment of the invention, the frequency of each word in the keyword set is first counted, then the frequency is multiplied by the preset sentiment weight factor corresponding to the word to obtain the product value, then the product values ​​of all words are accumulated, and finally the accumulated result is output as the numerical quantification result of the text unit.

[0125] Step 304: Combine each of the numerical quantization results based on time order to generate a text feature vector.

[0126] In this step, the combination operation refers to the process of connecting discrete values ​​into a vector along the time axis, preserving the temporal continuity of the evolution of emotions in the dialogue.

[0127] In this embodiment of the invention, the numerical quantization results of all text units are first extracted, then these results are arranged in chronological order according to the text units, then adjacent results are joined together, and finally a text feature vector of continuous time series is generated.

[0128] This invention captures changes in the rhythm of dialogue through time window segmentation, and combines emotional vocabulary weighted quantization to calibrate the emotional intensity of the text. Finally, it generates a text feature vector that preserves the semantic temporal regularity, which significantly improves the accuracy of recognizing professional fatigue expressions in customer service responses.

[0129] To eliminate feature conflicts caused by the temporal misalignment between voice tremor and text response, this step extracts a bimodal timestamp sequence, matches overlapping time points, synchronously extracts aligned feature points, and concatenates them into a fused feature. This invention provides a specific embodiment where step 104 involves aligning the emotion feature vector and the text feature vector using timestamps to generate a multimodal temporal fusion feature, specifically including the following steps:

[0130] Step 401: Extract the timestamp corresponding to each emotion feature point in the emotion feature vector to generate an emotion timestamp sequence.

[0131] In this step, the emotion feature point refers to the smallest data unit in the emotion feature vector, which includes the amplitude quantization value and waveform features at a specific timestamp.

[0132] In this embodiment of the invention, each data point in the emotion feature vector is first scanned to obtain its timestamp value, then all timestamp values ​​are collected and arranged in ascending order of time axis, and finally a continuous emotion timestamp sequence is generated.

[0133] Step 402: Extract the timestamp corresponding to each text feature point in the text feature vector to generate a text timestamp sequence.

[0134] In this step, text feature points refer to the basic data units in the text feature vector, which contain scalar values ​​of sentiment intensity at a specific timestamp.

[0135] In this embodiment of the invention, each data unit in the text feature vector is first parsed to read its timestamp value, then all timestamp values ​​are sorted according to their occurrence time, and finally a continuous text timestamp sequence is generated.

[0136] Step 403: Match the overlapping time points of the emotion timestamp sequence with the text timestamp sequence to generate matching timestamp pairs.

[0137] In this step, overlapping time points refer to time stamp value pairs in the dual-modal timestamp sequence that meet the time error tolerance range; the matching operation refers to the process of filtering valid time point pairs through tolerance comparison, including traversing the sequence and determining the tolerance threshold; the matched timestamp pair refers to the combination of timestamps generated after tolerance filtering, which serves as the benchmark for feature alignment.

[0138] In this embodiment of the invention, the emotional timestamp sequence and the text timestamp sequence are first traversed, then the time point combinations in the two sequences with a time difference less than a preset tolerance range are identified, and finally the time point pairs that meet the conditions are marked as matching timestamp pairs.

[0139] Step 404: Based on the matching timestamp pair, perform synchronous feature extraction operations on the emotion feature vector and the text feature vector respectively to generate aligned emotion feature points and aligned text feature points respectively.

[0140] In this step, the synchronous feature extraction operation refers to the action of locating and extracting corresponding feature points based on the matching timestamp, ensuring cross-modal temporal consistency; the aligning emotion feature points refers to the data points extracted from the emotion feature vector according to the matching timestamp, with the time reference synchronized with the text modality; the aligning text feature points refers to the data units extracted from the text feature vector according to the matching timestamp, with the time reference synchronized with the speech modality.

[0141] In this embodiment of the invention, firstly, the corresponding data point in the emotion feature vector is located based on the time value in the matching timestamp pair, and aligned emotion feature points are generated; secondly, the corresponding data unit in the text feature vector is located based on the same time value, and aligned text feature points are generated.

[0142] Step 405: Concatenate the aligned emotion feature points with the aligned text feature points to generate multimodal temporal fusion features.

[0143] In this step, the splicing operation refers to the process of merging bimodal feature points into composite data units based on the same timestamp, while preserving the original feature dimensions.

[0144] In this embodiment of the invention, each aligned emotion feature point is first merged with its corresponding aligned text feature point, then the merged composite data points are arranged in chronological order, and finally a multimodal temporal fusion feature is generated.

[0145] This invention establishes a unified time reference by using dual-modal timestamp tolerance matching, achieving millisecond-level precise alignment of speech tremor features and text emotion features, eliminating feature conflicts caused by cross-modal response delays, and significantly improving the temporal consistency of multimodal fusion features.

[0146] To address matching discrepancies caused by clock differences between cross-modal devices, this step calculates the absolute difference of all timestamps, filters candidate pairs within the tolerance range, and averages the results to generate hardware-level synchronized matching timestamp pairs. This invention provides a specific embodiment where step 403 matches the overlapping time points of the emotion timestamp sequence and the text timestamp sequence to generate matching timestamp pairs, specifically including the following steps:

[0147] Step 431: Calculate the absolute time difference between each emotion timestamp in the emotion timestamp sequence and each text timestamp in the text timestamp sequence to generate an absolute time difference set.

[0148] In this step, the emotion timestamp refers to the precise time stamp corresponding to the emotion feature point, which is derived from the clock record of the moment the voice tremor feature is generated; the text timestamp refers to the occurrence time value corresponding to the text feature point, which is based on the system clock calibration when the dialogue text is generated; the absolute time difference refers to the absolute value of the difference between the emotion timestamp and the text timestamp, reflecting the time offset of the bimodal data.

[0149] In this embodiment of the invention, each emotional timestamp in the emotional timestamp sequence is first traversed, then the absolute time difference between each emotional timestamp and all text timestamps in the text timestamp sequence is calculated, then all calculation results are summarized into a set, and finally an absolute time difference set containing all time difference values ​​is generated.

[0150] Step 432: Select target absolute time differences from the set of absolute time differences that are less than the preset time difference tolerance range, and generate a candidate time difference set.

[0151] In this step, the preset time difference tolerance range refers to the maximum allowed time deviation threshold, which is set according to the voice and text response delay statistics of customer service dialogue; the candidate time difference set refers to the effective time difference dataset after tolerance filtering, which contains alignable timestamp combinations.

[0152] In this embodiment of the invention, a preset time difference tolerance range threshold is first read, then each time difference value in the absolute time difference set is scanned, then time differences less than the threshold are selected, and finally time differences that meet the conditions are marked as candidate time difference sets.

[0153] Step 433: Calculate the average value of the emotion timestamp and text timestamp corresponding to each time difference in the candidate time difference set to generate a matching timestamp pair.

[0154] In this step, the average calculation refers to the operation of adding the two timestamps and dividing by two, which is used to generate a unified time base to eliminate device clock errors.

[0155] In this embodiment of the invention, firstly, the emotional timestamp and text timestamp corresponding to each time difference in the candidate time difference set are selected; secondly, the two timestamps are added together and divided by two to calculate the arithmetic mean; then, the calculation result is used as the time base value for the matching timestamp pair; finally, a matching timestamp pair with a unified time base is generated.

[0156] This invention locates potential alignment points by fully calculating the difference between bimodal timestamps, eliminates accidental timing offsets through tolerance screening, and then fuses the device clock differences by averaging the results, thereby achieving hardware-level synchronization of cross-modal time bases and achieving millisecond-level alignment accuracy between speech and text features.

[0157] To accelerate minute-level early warning response to the inflection point of emotional exhaustion, this step involves segmenting and fusing features to generate unit values, overlaying historical state updates to memory features, comparing the emotional baseline to generate probability distributions, detecting continuous trend markers for state transitions, and finally combining them into an analysis scheme. This invention provides a specific embodiment: Step 105, using a preset multimodal temporal perception model to perform state analysis processing on the multimodal temporal fusion features, generates an emotional analysis scheme for customer service dialogue emotional analysis, specifically including the following steps:

[0158] Step 501: Based on the time sequence, the multimodal temporal fusion feature is divided into analysis units of fixed time length, and the amplitude-weighted average value of all feature points in each analysis unit is calculated to generate unit fusion features.

[0159] In this step, segmentation refers to the process of cutting multimodal features at fixed time intervals, including timed triggering and equal-length slice generation; analysis unit refers to the fixed-duration data block generated by segmentation, which contains an aligned set of multimodal feature points; amplitude-weighted average refers to the arithmetic mean of the feature point amplitude value multiplied by the speech tremor weighting coefficient, reflecting the emotional intensity within the unit; unit fusion feature refers to the scalar value characterizing the overall emotional state of a single analysis unit, which is calculated by amplitude-weighted average.

[0160] In this embodiment of the invention, firstly, a time axis cutting operation is performed on the multimodal time-series fusion features at a fixed duration to generate analysis units. Secondly, the weighted sum of the amplitude values ​​of all feature points in each analysis unit multiplied by the weight coefficient is calculated, then divided by the total number of feature points, and finally the calculation result is output as the unit fusion feature.

[0161] Step 502: Linearly superimpose the unit fusion feature of the current analysis unit with the state memory feature of the previous analysis unit to generate the state memory feature of the current analysis unit.

[0162] In this step, state memory features refer to feature vectors carrying historical emotional information, which are generated by linearly superimposing current and historical features; the linear superposition operation refers to the operation of adding current features and historical features proportionally, preserving temporal dependencies.

[0163] In this embodiment of the invention, the state memory features stored in the previous analysis unit are first obtained, then the unit fusion features of the current analysis unit are multiplied by a preset coefficient and added to the previous state memory features, then a new feature vector is generated, and finally the vector is updated to the state memory features of the current analysis unit.

[0164] Step 503: Compare the similarity between the state memory features and the preset emotional benchmark feature set to generate an emotional tendency probability distribution.

[0165] In this step, the preset emotional baseline feature set refers to the pre-stored typical emotional state feature template, including customer service scenario baseline values ​​such as anger, calmness, and fatigue; the similarity comparison operation refers to the process of calculating the Euclidean distance between the state memory features and the baseline features and converting it into a probability; the emotional tendency probability distribution refers to the set of probability values ​​for each emotional category, which sums to 100% and identifies the current dominant emotion.

[0166] In this embodiment of the invention, each benchmark feature in the preset emotional benchmark feature set is first read, then the similarity score between the state memory feature and each benchmark feature is calculated, then the score is converted into a probability value, and finally an emotional tendency probability distribution containing the probability of each emotional category is generated.

[0167] Step 504: Perform trend consistency detection on the probability distribution of sentiment tendency in three consecutive analysis units to generate sentiment state transition markers.

[0168] In this step, the trend consistency detection operation refers to the process of analyzing whether the dominant emotional change direction is consistent across three consecutive probability distributions; the emotional state transition label refers to the symbolic encoding that identifies emotional transition events, such as the transition code from calm to anger.

[0169] In this embodiment of the invention, the probability distribution of sentiment tendency of three consecutive analysis units is first extracted, then the direction of change of dominant sentiment category between adjacent distributions is calculated, then the consistency of the direction of change is detected, and finally the sentiment state transition marker that identifies the state transition is output.

[0170] Step 505: Combine each of the emotional state transition markers based on time sequence to generate a sentiment analysis scheme for conducting sentiment analysis of customer service dialogues.

[0171] In this step, the combination operation refers to the process of integrating discrete markers into a complete analysis report in chronological order.

[0172] In this embodiment of the invention, all sentiment state transition markers are first arranged in chronological order, then the marker sequence is converted into a structured data format, and finally an executable sentiment analysis scheme is generated.

[0173] This invention captures emotional evolution segments through temporal segmentation, tracks long-term emotional trends by combining state memory mechanisms, accurately locates the inflection point of emotional exhaustion through probability distribution comparison and trend consistency detection, and finally outputs an emotion analysis solution that can guide human resource scheduling, significantly shortening the early warning and response speed of emotional crises for high-load customer service.

[0174] Figure 2 This invention provides a schematic diagram of the structure of a customer service dialogue sentiment analysis system based on multimodal temporal awareness, as shown in the embodiment of the invention. Figure 2 As shown, the system includes:

[0175] The focusing module 21 is used to acquire physical vibration features in customer service voice through a preset acoustic sensor array, and to focus the physical vibration features to generate noise-reduced voice physical vibration features.

[0176] The conversion module 22 is used to perform feature conversion on the physical tremor features of the denoised speech based on a preset real-time emotional state classification circuit to generate an emotional feature vector.

[0177] Quantization module 23 is used to acquire the time-series dialogue text during the customer service dialogue process, perform quantization processing on the time-series dialogue text, and generate a text feature vector.

[0178] Alignment module 24 is used to perform timestamp alignment processing on the emotion feature vector and the text feature vector to generate multimodal temporal fusion features;

[0179] The analysis module 25 is used to perform state analysis processing on the multimodal temporal fusion features through a preset multimodal temporal perception model, and generate a sentiment analysis scheme for conducting sentiment analysis of customer service dialogues.

[0180] Figure 2The aforementioned customer service dialogue sentiment analysis system based on multimodal temporal awareness can execute... Figure 1 The implementation principle and technical effects of the customer service dialogue sentiment analysis method based on multimodal temporal awareness described in the illustrated embodiment will not be repeated here. The specific methods by which each module and unit of the customer service dialogue sentiment analysis system based on multimodal temporal awareness in the above embodiments are performed have been described in detail in the embodiments related to this method, and will not be elaborated upon here.

[0181] In one possible design, Figure 2 The customer service dialogue sentiment analysis system based on multimodal temporal awareness shown in the embodiment can be implemented as a computing device, such as... Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32;

[0182] The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are invoked and executed by the processing component 32.

[0183] The processing component 32 is used to: acquire physical tremor features in customer service voice through a preset acoustic sensor array, focus on the physical tremor features to generate denoised voice physical tremor features; perform feature transformation on the denoised voice physical tremor features based on a preset real-time emotion state classification circuit to generate an emotion feature vector; acquire the temporal dialogue text during the customer service conversation, quantize the temporal dialogue text to generate a text feature vector; perform timestamp alignment processing on the emotion feature vector and the text feature vector to generate multimodal temporal fusion features; and perform state analysis processing on the multimodal temporal fusion features through a preset multimodal temporal perception model to generate an emotion analysis scheme for performing emotion analysis of customer service conversations.

[0184] The processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above-described method. Alternatively, the processing component may be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described method.

[0185] Storage component 31 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0186] Of course, computing devices may also include other components, such as input / output interfaces, display components, communication components, etc.

[0187] Input / output interfaces provide interfaces between processing components and peripheral interface modules, which can be output devices, input devices, etc.

[0188] The communication components are configured to facilitate wired or wireless communication between computing devices and other devices.

[0189] The computing device can be a physical device or an elastic computing host provided by a cloud computing platform. In this case, the computing device can refer to a cloud server, and the aforementioned processing components, storage components, etc., can be basic server resources rented or purchased from the cloud computing platform.

[0190] This invention also provides a computer storage medium storing a computer program, which, when executed by a computer, can perform the above-described functions. Figure 1 The embodiment shown is a customer service dialogue sentiment analysis method based on multimodal temporal awareness.

[0191] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0192] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0193] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-modal time-series perception based customer service dialogue sentiment analysis method, characterized in that, include: The process involves acquiring physical vibration features in customer service voice messages using a preset acoustic sensor array, focusing on these physical vibration features to generate denoised voice physical vibration features, and then: acquiring customer service voice signals using sensor units in the preset acoustic sensor array; extracting oscillation components with an oscillation frequency range of 50Hz to 150Hz from the unit signals of each sensor unit to generate initial physical vibration features for each sensor unit; calculating the phase difference between each initial physical vibration feature; locating the spatial source direction of the customer service voice signal based on the phase difference; assigning oscillation component weights to each sensor unit based on the spatial source direction to generate weighted oscillation components; aggregating the weighted oscillation components to generate a aggregated oscillation signal; and selecting continuous oscillation segments with amplitude fluctuations greater than a preset fluctuation threshold from the aggregated oscillation signal, marking these continuous oscillation segments as denoised voice physical vibration features. Based on a preset real-time emotional state classification circuit, the physical vibration features of the denoised speech are transformed to generate an emotional feature vector. The process involves acquiring the time-series dialogue text during customer service conversations, quantifying the text, and generating a text feature vector. This includes: segmenting the time-series dialogue text according to a preset time window to generate multiple text units arranged chronologically; extracting a set of words containing sentiment from each text unit to generate a keyword set for each text unit; multiplying the frequency of each keyword in each keyword set with a preset sentiment weight factor, summing the product results to generate a numerical quantization result for each text unit; and combining each numerical quantization result based on chronological order to generate a text feature vector. The process of aligning the emotion feature vector and the text feature vector with timestamps to generate multimodal temporal fusion features includes: extracting the timestamp corresponding to each emotion feature point in the emotion feature vector to generate an emotion timestamp sequence; extracting the timestamp corresponding to each text feature point in the text feature vector to generate a text timestamp sequence; matching the overlapping timestamps of the emotion timestamp sequence and the text timestamp sequence to generate matching timestamp pairs; performing synchronous feature extraction operations on the emotion feature vector and the text feature vector based on the matching timestamp pairs to generate aligned emotion feature points and aligned text feature points respectively; and concatenating the aligned emotion feature points and the aligned text feature points to generate multimodal temporal fusion features. The multimodal temporal fusion features are analyzed and processed using a preset multimodal temporal perception model to generate a sentiment analysis scheme for conducting sentiment analysis in customer service dialogues.

2. The method of claim 1, wherein, Based on a preset real-time emotion state classification circuit, the physical tremor features of the denoised speech are transformed to generate an emotion feature vector, including: Based on a preset real-time emotional state classification circuit, the denoised speech physical tremor features are divided into multiple continuous feature segments of equal time length. Calculate the maximum and minimum amplitude values ​​for each of the continuous feature segments, and perform a difference operation on the maximum and minimum amplitude values ​​to generate the amplitude fluctuation intensity of the continuous feature segments; Based on the time sequence of the amplitude fluctuation intensity, three consecutive amplitude fluctuation intensities are selected, and the slope of the change of the three consecutive amplitude fluctuation intensities is calculated to generate a dynamic change trend. Based on the dynamic change trend, weighting factors are assigned to the three consecutive feature segments corresponding to the three consecutive amplitude fluctuation intensities to generate weighted feature segments; Each of the weighted feature segments is connected to generate an emotion feature vector.

3. The method of claim 1, wherein, Matching the overlapping time points of the emotion timestamp sequence with the text timestamp sequence to generate matching timestamp pairs includes: Calculate the absolute time difference between each emotion timestamp in the emotion timestamp sequence and each text timestamp in the text timestamp sequence to generate an absolute time difference set; From the set of absolute time differences, select target absolute time differences whose absolute time differences are less than a preset time difference tolerance range to generate a candidate time difference set; The average value of the emotion timestamp and text timestamp corresponding to each time difference in the candidate time difference set is calculated to generate a matching timestamp pair.

4. The method according to claim 1, characterized in that, The multimodal temporal fusion features are processed using a pre-defined multimodal temporal perception model to generate a sentiment analysis scheme for customer service dialogue sentiment analysis, including: Based on the time sequence, the multimodal temporal fusion features are segmented into analysis units of fixed time length, and the amplitude-weighted average of all feature points in each analysis unit is calculated to generate unit fusion features; The unit fusion feature of the current analysis unit is linearly superimposed with the state memory feature of the previous analysis unit to generate the state memory feature of the current analysis unit; The state memory features are compared with a preset set of emotional benchmark features to generate an emotional tendency probability distribution. A trend consistency detection is performed on the probability distribution of sentiment tendency in three consecutive analysis units to generate sentiment state transition markers; Each emotional state transition marker is combined based on time sequence to generate a sentiment analysis scheme for conducting sentiment analysis of customer service conversations.

5. A customer service dialogue sentiment analysis system based on multimodal temporal awareness, applied to the customer service dialogue sentiment analysis method based on multimodal temporal awareness as described in any one of claims 1-4, characterized in that, include: The focusing module is used to acquire physical vibration features in customer service voice through a preset acoustic sensor array, and to focus the physical vibration features to generate noise-reduced voice physical vibration features. The conversion module is used to perform feature conversion on the physical tremor features of the denoised speech based on a preset real-time emotional state classification circuit, and generate an emotional feature vector. The quantization module is used to acquire the time-series dialogue text during customer service conversations, perform quantization processing on the time-series dialogue text, and generate a text feature vector. The alignment module is used to perform timestamp alignment processing on the emotion feature vector and the text feature vector to generate multimodal temporal fusion features; The analysis module is used to perform state analysis processing on the multimodal temporal fusion features through a preset multimodal temporal perception model, and generate a sentiment analysis scheme for conducting sentiment analysis of customer service dialogues.

6. A computing device, characterized in that, It includes a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are invoked and executed by the processing component to implement a customer service dialogue sentiment analysis method based on multimodal temporal awareness as described in any one of claims 1 to 4.

7. A computer storage medium, characterized in that, The system contains a computer program that, when executed by a computer, implements a customer service dialogue sentiment analysis method based on multimodal temporal awareness as described in any one of claims 1 to 4.

Images