A Method and System for Sentiment Analysis of Call Center Dialogues Integrating Voice and Text

Abstract

This invention provides a method and system for call center dialogue sentiment analysis that integrates speech and text, belonging to the field of speech signal processing technology. The method includes: acquiring speech signals and corresponding transcribed text from the call center; extracting tone, speech rate, intensity, and pause sentiment features from the speech signals; simultaneously performing deep language analysis on the transcribed text to extract context-dependent semantic sentiment features; conducting cross-modal correlation analysis on the speech and semantic sentiment features; and generating temporal correction coefficients for feature alignment by constructing a correspondence analysis framework between feature sequences. This invention integrates linguistic sentiment information from speech and text, enabling more accurate and comprehensive identification of sentiment in call center dialogues, providing a reliable sentiment analysis basis for improving customer satisfaction and efficiently handling customer requests.

Description

Technical Field

[0001] This invention relates to the field of speech signal processing technology, and in particular to a method and system for analyzing the sentiment of call center conversations that integrates speech and text. Background Technology

[0002] In after-sales consultations at e-commerce platform call centers, traditional text sentiment analysis technology mostly judges user emotions by identifying keywords and semantic logic. This may fail to capture the subtle nuances of tone implied in the voice, which are often crucial for emotional expression in after-sales consultations. For example, a user might send a text message to online customer service saying, "The size of the clothes I bought doesn't fit, please exchange it." While the text may not contain explicit negative words, a traditional text analysis system might classify the user's emotion as neutral. However, based on the actual voice interaction, the user's tone might be hurried, accompanied by slight impatience and pauses, indicating mild dissatisfaction. This difference in emotion can affect the priority of customer service responses. If treated as neutral, customer service might follow the standard procedure, delaying timely reassurance of the user's dissatisfaction. Capturing tone information allows for priority customer service follow-up, reducing the probability of escalating complaints.

[0003] Furthermore, when there is semantic ambiguity in the dialogue, relying solely on voice features is insufficient to accurately interpret emotions, potentially leading to misjudgments where tone contradicts meaning. For example, a user inquires about after-sales service due to product damage, saying, "Your packaging is too sturdy; I spent half an hour unpacking it, and the cup inside still broke." From a voice feature perspective, the user's tone is slightly high-pitched and has a teasing pause when saying "too sturdy." A traditional voice analysis system might mistakenly interpret this tone as mildly positive. However, considering the textual semantics, "sturdy" here is ironic, actually expressing that the packaging is excessive and useless, with an overall emotion of moderate dissatisfaction. If relying solely on voice analysis, customer service might misunderstand the user's emotions, failing to provide a timely and effective compensation solution, thus exacerbating the user's dissatisfaction. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a method and system for call center dialogue sentiment analysis that integrates voice and text, thereby improving the accuracy and comprehensiveness of sentiment analysis and helping call centers to accurately grasp the emotional state of customers and adjust customer service communication strategies in a timely manner.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0006] Firstly, a method for sentiment analysis of call center dialogues that integrates voice and text, the method comprising:

[0007] The system acquires voice signals and corresponding transcribed text from call centers, extracts tone, speech rate, volume, and pause emotion features from the voice signals, and performs deep language analysis on the transcribed text to extract context-related semantic and emotional features.

[0008] Cross-modal correlation analysis is performed on speech sentiment features and semantic sentiment features. By constructing a correspondence analysis framework between feature sequences, temporal correction coefficients for feature alignment are generated.

[0009] Using time-series correction coefficients, multimodal synchronous processing is performed on speech emotion features and semantic emotion features to identify and correct time-series misalignment and semantic inconsistency caused by slips of the tongue, pauses and transcription errors, and generate a consistent multimodal feature sequence.

[0010] Based on multimodal feature sequences, a cross-modal attention mechanism is used to achieve deep fusion of speech sentiment features and semantic sentiment features. According to the confidence and semantic consistency of each modality, the influence of different modalities in sentiment analysis is dynamically adjusted to generate a joint representation vector including global sentiment tendency.

[0011] The joint representation vector is input into the sentiment classifier to obtain the final sentiment category and intensity.

[0012] Furthermore, cross-modal correlation analysis is performed on speech sentiment features and semantic sentiment features. By constructing a correspondence analysis framework between feature sequences, temporal correction coefficients for feature alignment are generated, including:

[0013] Based on speech emotion feature sequences and semantic emotion feature sequences, and combined with the timestamp information of speech frames and text words, a fine-grained temporal correspondence between the two modal feature sequences is established.

[0014] Based on fine-grained temporal correspondence, a cross-modal feature alignment framework is constructed to organize and map two modal features on a shared temporal dimension.

[0015] Based on the cross-modal feature alignment framework, the dynamic similarity measure and correlation coefficient between speech emotion features and semantic emotion features are calculated within the same time interval.

[0016] Based on the dynamic similarity metric and correlation coefficient, a set of temporal correction coefficients for feature alignment is generated.

[0017] Furthermore, based on fine-grained temporal correspondences, a cross-modal feature alignment framework is constructed to organize and map features of two modalities along a shared temporal dimension, including:

[0018] Based on the temporal correction coefficient, the dynamic alignment path between each frame in the speech feature sequence and each word unit in the text feature sequence is calculated to establish a temporal mapping relationship from speech to text.

[0019] Based on the dynamic alignment path, the feature vector of each frame in the speech feature sequence is mapped to the timestamp position of the corresponding word unit in the text feature sequence, so as to realize the projection and alignment of cross-modal features under a unified temporal coordinate.

[0020] Under a unified temporal coordinate system, the mapped speech features and corresponding text word unit features are aligned and concatenated one by one, and recombined into a synchronous multimodal fusion feature sequence.

[0021] Furthermore, using time-series correction coefficients, multimodal synchronous processing is performed on speech and semantic emotion features to identify and correct time-series misalignments and semantic inconsistencies caused by slips of the tongue, pauses, and transcription errors, generating a consistent multimodal feature sequence, including:

[0022] The temporal correction coefficients are applied to the speech emotion feature sequence and the semantic emotion feature sequence respectively to establish the mapping relationship between the two modal features;

[0023] Based on the mapping relationship, the final alignment path is determined by calculating the cumulative distance between the two modal feature sequences, and the temporal scale of the speech emotion feature sequence and the semantic emotion feature sequence is adjusted to achieve temporal alignment of the feature sequences.

[0024] Based on temporal alignment, detect and correct semantic inconsistencies caused by slips of the tongue, repetitions, and automatic speech recognition errors in the speech signal;

[0025] The corrected speech emotion features and semantic emotion features are fused to generate a temporally consistent and coordinated multimodal feature sequence.

[0026] Furthermore, based on multimodal feature sequences, a cross-modal attention mechanism is employed to achieve deep fusion of speech sentiment features and semantic sentiment features. The influence of different modalities in sentiment analysis is dynamically adjusted according to the confidence and semantic consistency of each modality, generating a joint representation vector including global sentiment tendency, comprising:

[0027] Cross-modal temporal alignment is performed on the speech emotion feature sequence and the text emotion feature sequence to obtain the aligned multimodal feature sequence;

[0028] The aligned multimodal feature sequences are input into the cross-modal attention mechanism. Bidirectional attention calculations are performed using speech features as queries and text features as keys, and text features as queries and speech features as keys, respectively, to realize the interaction modeling between modalities and extract deep semantic association features to obtain cross-modal fusion features.

[0029] Based on cross-modal fusion features, the acoustic confidence index of speech features and the semantic consistency index of text features are calculated separately. The importance of speech modality and text modality in emotion recognition is dynamically determined according to the two types of indices.

[0030] Based on modal importance, cross-modal fusion features are adaptively fused to generate a joint sentiment representation vector that includes global contextual sentiment semantic information.

[0031] Furthermore, the joint representation vector is input into the sentiment classifier to obtain the final sentiment category and intensity, including:

[0032] The joint representation vector is input into a pre-trained sentiment classifier, and the probability distribution of sentiment categories is generated through multi-layer neural network calculation.

[0033] Based on the probability distribution of sentiment categories, the intensity of sentiment is calculated through regression branches;

[0034] The probability distribution of sentiment categories is combined with the sentiment intensity values ​​to obtain the final sentiment analysis results, which include sentiment category labels and corresponding intensity values.

[0035] Furthermore, the sentiment categories include neutral, positive, and negative multi-level labels.

[0036] Secondly, a call center dialogue sentiment analysis system that integrates voice and text includes:

[0037] The feature extraction module is used to acquire the voice signal and corresponding transcribed text of the call center, extract tone, speech rate, volume and pause emotion features from the voice signal, and perform deep language analysis on the transcribed text to extract semantic emotion features related to the context.

[0038] The association analysis module is used to perform cross-modal association analysis on speech sentiment features and semantic sentiment features. By constructing a correspondence analysis framework between feature sequences, it generates temporal correction coefficients for feature alignment.

[0039] The synchronization correction module is used to perform multimodal synchronous processing on speech emotion features and semantic emotion features using time-series correction coefficients, identify and correct time-series misalignment and semantic inconsistency caused by slips of the tongue, pauses and transcription errors, and generate a consistent multimodal feature sequence.

[0040] The fusion representation module is used to achieve deep fusion of speech sentiment features and semantic sentiment features based on multimodal feature sequences and cross-modal attention mechanism. It dynamically adjusts the influence of different modalities in sentiment analysis according to the confidence and semantic consistency of each modality, and generates a joint representation vector including global sentiment tendency.

[0041] The classification decision module is used to input the joint representation vector into the sentiment classifier to obtain the final sentiment category and intensity.

[0042] Thirdly, a computing device, comprising:

[0043] One or more processors;

[0044] A storage device for storing one or more programs that, when executed by one or more processors, cause the one or more processors to implement the method.

[0045] Fourthly, a computer-readable storage medium storing a program that, when executed by a processor, implements the method.

[0046] The above-described solution of the present invention has at least the following beneficial effects:

[0047] This system extracts acoustic emotional features such as intonation and speech rate from speech signals, as well as contextual semantic emotional features from transcribed text, achieving bimodal information complementarity and avoiding information loss caused by single modality. Simultaneously, it generates temporal correction coefficients through cross-modal association analysis, establishing a fine-grained mapping between speech frames and text units to correct temporal misalignments and semantic contradictions caused by slips of the tongue, pauses, and transcription errors. Furthermore, it dynamically adjusts bimodal weights using a cross-modal attention mechanism, resulting in higher sentiment analysis accuracy than single-modal or simple splicing fusion schemes. Addressing common issues in call center dialogues such as slips of the tongue, repetitions, and automatic speech recognition errors, it adjusts the time scale of feature sequences through temporal correction and actively identifies and corrects these errors using semantic consistency detection. It also supports dual output of neutral, positive, and negative multi-level sentiment labels and sentiment intensity values, clearly identifying customer sentiment tendencies and quantifying emotional intensity, meeting the practical needs of customer service for segmented customer emotion management and improving applicability in complex dialogue scenarios.

[0048] For customer service, it can accurately capture changes in customer emotions in real time, assisting customer service representatives in dynamically adjusting communication strategies, reducing customer complaint rates and invalid call duration, and improving the success rate of single service. For management, based on the analysis results of emotion categories and intensities, it can review the effectiveness of customer service communication in batches, providing data support for customer service training and script optimization, and promoting service standardization. At the same time, through the accumulation of emotion data, it can also optimize products or services in reverse. The cross-modal feature alignment framework, the time-series correction coefficient calculation logic, and the modal weight dynamic adjustment mechanism can be adapted to different industry scenarios by fine-tuning the calculation parameters of acoustic confidence and semantic consistency. Attached Figure Description

[0049] Figure 1This is a flowchart illustrating a call center dialogue sentiment analysis method that integrates voice and text, as provided in an embodiment of the present invention.

[0050] Figure 2 This is a schematic diagram of a call center dialogue sentiment analysis system that integrates voice and text, provided by an embodiment of the present invention. Detailed Implementation

[0051] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0052] like Figure 1 As shown, an embodiment of the present invention proposes a call center dialogue sentiment analysis method that integrates voice and text, the method comprising the following steps:

[0053] Step 1: Obtain the voice signal and corresponding transcribed text from the call center. Extract intonation, speech rate, volume and pause emotional features from the voice signal. At the same time, perform deep language analysis on the transcribed text to extract semantic and emotional features related to the context.

[0054] Step 2: Perform cross-modal correlation analysis on speech emotion features and semantic emotion features. By constructing a correspondence analysis framework between feature sequences, generate temporal correction coefficients for feature alignment.

[0055] Step 3: Using time-series correction coefficients, multimodal synchronous processing is performed on speech emotion features and semantic emotion features to identify and correct time-series misalignment and semantic inconsistency caused by slips of the tongue, pauses and transcription errors, and generate a consistent multimodal feature sequence.

[0056] Step 4: Based on the multimodal feature sequence, a cross-modal attention mechanism is used to achieve deep fusion of speech emotion features and semantic emotion features. According to the confidence and semantic consistency of each modality, the influence of different modalities in sentiment analysis is dynamically adjusted to generate a joint representation vector including global sentiment tendency.

[0057] Step 5: Input the joint representation vector into the sentiment classifier to obtain the final sentiment category and intensity.

[0058] In this embodiment of the invention, by simultaneously acquiring speech signals and corresponding transcribed text, and extracting two types of features, the system captures both the emotional dynamics of speech, such as intonation (e.g., rising intonation reflects excitement, rapid speech reflects anxiety, loud voice conveys dissatisfaction), pauses (e.g., long pauses suggest hesitation), and the semantic logic features of the text in conjunction with the context (e.g., negative semantics indicating unresolved issues after multiple feedbacks). These two types of features complement each other, avoiding the problems of relying solely on speech being susceptible to noise interference and relying solely on text losing information about emotional intensity. This makes the information foundation for sentiment analysis more complete. By performing cross-modal correlation analysis on speech and semantic sentiment features, a feature sequence correspondence framework is constructed and temporal correction coefficients are generated. This clarifies the matching relationship between speech frames and text words in the temporal dimension, such as mapping speech pauses to semantic scenarios of sentence intervals in the text. This avoids emotional misalignment caused by the asynchronous rhythm of speech delivery and the speed of text transcription, such as speech expressing anger but the text not corresponding to related complaints. This removes the temporal obstacles for feature fusion and improves the accuracy of the analysis foundation.

[0059] When using temporal correction coefficients to synchronously process bimodal features, it can proactively identify common slips of the tongue in call center dialogues, such as saying "satisfied" instead of "dissatisfied," pauses leading to transcription omissions or errors, and transing "irritable" into "irritable." It then specifically corrects these temporal misalignments, such as adjusting the correspondence between speech frames and text words and semantic inconsistencies, and correcting text semantic deviations caused by slips of the tongue. Ultimately, it generates a consistent multimodal feature sequence, reducing the interference of erroneous data on analysis and making the features more closely match the emotional expression of real dialogue. A cross-modal attention mechanism is employed to allow full interaction between speech and semantic features, such as highlighting the correlation between angry tones and complaining text through attention weights. Simultaneously, it combines the confidence levels of each modality, such as increasing the weight of speech features when the speech is clear and increasing the weight of text features and semantic consistency when the text semantics are clear. In scenarios like smiling while expressing dissatisfaction, the system can determine which modality is more credible and dynamically adjust the influence of both modalities. The resulting joint representation vector integrates global sentiment tendencies, avoiding analysis biases caused by single-modal defects such as unclear speech or ambiguous text. It can also flexibly optimize feature contributions based on the dialogue scenario, making sentiment representation more accurate. When the joint representation vector is input into the sentiment classifier, it can not only clearly distinguish sentiment categories such as neutral, positive, and negative, but also output the corresponding sentiment intensity, such as slightly negative or severely negative. Compared to analyses that only judge sentiment tendencies, this result allows customer service representatives to more accurately grasp the customer's emotional state, such as identifying when a customer's dissatisfaction escalates to severe anger, and then adjust communication strategies accordingly, such as patiently explaining when there is slight dissatisfaction and prioritizing the resolution of core issues when there is severe anger, thus improving the adaptability and effectiveness of customer service.

[0060] In a preferred embodiment of the present invention, step 1 above, which involves acquiring the voice signal and corresponding transcribed text from the call center, extracting intonation, speech rate, volume, and pause emotion features from the voice signal, and simultaneously performing deep language analysis on the transcribed text to extract context-related semantic and emotional features, may include:

[0061] In this embodiment of the invention, the voice signal during a call is captured in real time or offline using recording equipment in the call center, such as agent microphones or call recording systems. This ensures that the parameters of the acquisition equipment, such as sampling rate and bit depth, are uniform (common sampling rate is 16kHz and bit depth is 16bit), avoiding voice signal distortion due to equipment differences. The storage format of the voice signal is also recorded, such as WAV or MP3, for subsequent signal analysis based on the format. Speech-to-text (ASR) technology is used to convert the acquired voice signal into text segment by segment. First, the voice signal is preprocessed, such as noise reduction and removal of silent segments, to reduce the impact of background noise on transcription accuracy. Then, the acoustic mechanism of the ASR system matches voice features with articulatory units, and the language mechanism corrects grammatical and semantic errors, generating transcribed text aligned with the timeline of the voice signal. Each sentence and word in the text corresponds to a specific time segment of the voice signal. Finally, a portion of the text is manually sampled to correct transcription errors, such as recognition deviations caused by dialects or technical terms, ensuring accurate correspondence between the text and the voice signal. The signal undergoes pre-emphasis processing, using a high-pass filter to enhance high-frequency signals and compensate for the attenuation of high-frequency components during propagation. Framing is also employed, dividing the continuous speech signal into frames of 10-30ms in length, with an inter-frame overlap rate of 50%-75% to avoid loss of frame boundary information. Windowing is then used, employing Hamming windows and other window functions to reduce spectral leakage at frame edges. For each pre-processed frame, the fundamental frequency (F0), the fundamental frequency of vocal cord vibration and a core indicator of pitch, is calculated using autocorrelation or cepstral method. The autocorrelation method calculates the signal's time delay. The correlation of the signals is analyzed, and the delay value with the highest correlation is found and converted into the fundamental frequency. The cepstral method separates the fundamental frequency information from the cepstral by taking the logarithm of the signal spectrum and performing an inverse Fourier transform. The dynamic changes of the fundamental frequency in the entire speech segment are statistically analyzed, including the mean fundamental frequency, which reflects the overall pitch, such as a higher mean fundamental frequency when the emotion is excited; the standard deviation of the fundamental frequency, which reflects the amplitude of pitch fluctuation, such as greater pitch fluctuation when angry; and the fundamental frequency trend, which calculates the direction of fundamental frequency change over time through linear fitting or sliding window, such as an upward trend in the fundamental frequency when joyful and a downward trend when sad.

[0062] Silence detection is performed on the speech signal by setting energy thresholds (e.g., below 10% of the average energy of the overall speech signal) and duration thresholds (e.g., exceeding 50ms continuously). This distinguishes between silent segments and valid speech segments, removing invalid silences at the beginning, end, and middle, retaining only valid segments containing spoken content, and calculating the total duration of valid speech segments. Simultaneously, the total number of words or characters in the transcribed text is counted, depending on the language type (e.g., characters for Chinese, words for English). Meaningless filler words such as "um," "ah," and "this" are excluded using a preset dictionary. The speech rate is obtained by dividing the total number of words or characters within the valid speech segments by the total duration of the valid speech segments. Furthermore, the fluctuation characteristics of the speech rate can be calculated by using a sliding window (e.g., a 1-second window duration) to calculate the local speech rate within each window, and then calculating the standard deviation of the local speech rate to reflect the stability of the speaker's speech rate; for example, speech rate fluctuations are larger when the speaker is nervous.

[0063] For each frame of the speech signal after framing, the energy of the signal within the frame is calculated. This is done by summing the squares of the amplitudes at each sampling point within the frame and then taking the average value. The energy level directly corresponds to the sound intensity, i.e., the strength or loudness of the sound. The energy characteristics of the entire effective speech segment are statistically analyzed, including the mean energy, which reflects the overall sound intensity level (e.g., the mean energy is higher when angry and lower when sad), the maximum and minimum energy values, which reflect the dynamic range of sound intensity (e.g., the dynamic range is larger when emotionally agitated), and the rate of change of energy. The speed of change of sound intensity is calculated by dividing the difference in energy between adjacent frames by the time interval (e.g., the sound intensity rises rapidly when surprised). At the same time, strong and weak segments can be divided by energy thresholds, and the proportion of strong segments in the effective speech segment is statistically analyzed (e.g., the proportion of strong segments is higher when confident).

[0064] Based on the silence detection results of the speech signal, pauses between valid speech segments are filtered out, excluding silence segments at the beginning and end. Pauses are categorized by duration: short pauses (0.2-0.5 seconds, commonly seen during brief breaths within sentences), medium pauses (0.5-1 seconds, commonly seen between clauses), and long pauses (over 1 second, commonly seen during thinking, hesitation, or topic changes). Key indicators of pauses are statistically analyzed, including the total number of pauses (more frequent pauses when nervous or hesitant), the percentage of pauses of different durations (a high percentage of long pauses may indicate unclear thinking), the average pause duration (the overall average pause time, longer average pauses when sad), and the distribution of pause locations. Corresponding to the transcribed text, it is determined whether pauses occur in grammatically correct positions, such as commas or periods. Frequent pauses in unmarked parts of sentences may indicate emotional instability.

[0065] The transcribed text is formatted and standardized by removing special symbols, correcting typos, and unifying capitalization. Simultaneously, word segmentation is performed: Chinese sentences are broken down into individual words using tools like Jieba, while English sentences are segmented directly using spaces, and stop words are removed. Based on the timeline correspondence of the transcribed text, the dialogue structure of the call is reconstructed, distinguishing between agent and customer texts. Text is further segmented into independent sentences according to sentence boundaries such as periods, question marks, and exclamation marks, clarifying the contextual logical relationships between sentences, such as cause and effect, contrast, and progression. Connective words are used to determine these relationships; for example, "because… therefore…" indicates cause and effect, while "but" indicates contrast. Based on the call center's business scenario, a domain keyword dictionary is constructed. Domain keywords are extracted from the text through dictionary matching, and the frequency and location of keyword occurrences are statistically analyzed. A general sentiment dictionary and a contextual sentiment dictionary are introduced to extract sentiment words from the text, and the sentiment tendency and intensity of each sentiment word are labeled. For negative sentences, the negation word is identified, and the tendency of subsequent sentiment words is reversed.

[0066] Based on the text's sentence segmentation results, we analyze the semantic relationships between adjacent sentences and contextual sentences. This is done through conjunctions, referential relationships (e.g., "it"), the preceding object of the question, and topic consistency (e.g., whether the discussion revolves around the same business keyword). We determine semantic logical coherence; for example, if a customer first mentions a logistics delay, and subsequent sentences revolve around when delivery will occur, the semantics are coherent. A sudden shift to poor customer service indicates a semantic transition, possibly accompanied by emotional changes. We statistically analyze the total frequency and intensity of positive and negative sentiment words in the text to calculate the overall sentiment tendency. If the total intensity of negative sentiment is significantly higher than positive, the overall tendency is negative. We also analyze changes in contextual sentiment tendency, such as from neutral consultation to negative realization of an unsolvable problem, and then to positive compensation, to determine if there are abrupt changes in emotion. For example, a sudden shift from calm to anger might correspond to a change in tone or intensity in speech. Finally, we calculate the sentiment tendency score using the formula: (Total intensity of positive sentiment words - Total intensity of negative sentiment words) / (Total intensity of positive sentiment words + Total intensity of negative sentiment words + ...). 1) Add 1 to avoid a denominator of 0. The closer the score is to 1, the stronger the positive tendency; the closer it is to -1, the stronger the negative tendency; near 0 is neutral. At the same time, count the proportion of negative sentiment words in the text. If the proportion exceeds 40%, it can be judged as high negative risk. Count the proportion of complex sentence structures in the text, such as long sentences and multiple clauses. A high proportion of complex sentence structures may reflect the speaker's rigorous expression, such as the agent explaining rules or emotional suppression, such as the customer deliberately organizing their language to complain. In addition, analyze the co-occurrence relationship between domain keywords and sentiment words, such as the co-occurrence of overdue and anger, and the co-occurrence of refund and satisfaction. Count the frequency of co-occurrence. Combinations with high co-occurrence frequency can be used as contextual sentiment features. For example, the combination of overdue and anger corresponds to the negative emotions of customers due to repayment problems.

[0067] In a preferred embodiment of the present invention, step 2 above, which involves performing cross-modal correlation analysis on speech emotion features and semantic emotion features, and generating temporal correction coefficients for feature alignment by constructing a correspondence analysis framework between feature sequences, may include:

[0068] Step 220: Based on the speech emotion feature sequence and semantic emotion feature sequence, and combined with the timestamp information of speech frames and text words, establish a fine-grained temporal correspondence between the two modal feature sequences. Specifically, this includes: first, sorting out the extracted speech emotion features, including the sequence data corresponding to intonation, speech rate, intensity, and pauses. For intonation features, according to the speech frame order, each frame is 10-30ms, with 50%-75% overlap between frames. Arrange the fundamental frequency mean, standard deviation, and trend value calculated for each frame in sequence to form an intonation feature sequence. Each sequence element corresponds to a specific frame timestamp, such as the first frame corresponding to 0-20ms. The second frame corresponds to 10-30ms, and so on. For speech rate features, the effective speech segments are divided by sliding windows, such as 1 second per window. The local speech rate values ​​calculated for each window are arranged in chronological order to form a speech rate feature sequence. Each element corresponds to the start and end timestamps of the window. For example, the first window corresponds to 0-1 seconds, and the second window corresponds to 0.5-1.5 seconds. For intensity features, the energy values ​​calculated for each frame are arranged in the order of the frames to form an intensity feature sequence, with each element corresponding to a frame timestamp. For pause features, the duration, start and end timestamps of each pause are organized into a pause feature sequence according to the order in which the pauses are detected.

[0069] For the semantic sentiment features extracted from the transcribed text, the results are first split into sentences, using periods, question marks, and exclamation marks as boundaries. The sentiment tendency score, the sum of sentiment word intensity, and the co-occurrence frequency of domain keywords and sentiment words corresponding to each sentence are organized into a basic sequence. Then, combined with the text word timestamps generated by ASR, each word corresponds to a specific time period in the speech signal, such as "Hello" corresponding to 0.3-0.8 seconds and "Please ask" corresponding to 0.9-1.4 seconds. This is further refined to the word level. For each word, it is determined whether it is a sentiment word or a domain keyword. If it is a sentiment word, its sentiment tendency and intensity are marked. If it is a domain keyword, its association with surrounding sentiment words is recorded. The words are arranged according to their order in the text to form a word-level semantic sentiment feature sequence, with each element corresponding to the precise timestamp of the word.

[0070] Using the total time axis of the speech signal as a benchmark, the timestamps of the speech emotion feature sequence elements are compared one by one with the timestamps of the semantic emotion feature sequence elements. For features corresponding to speech frames, such as intonation and intensity, the overlap ratio between the timestamp of each frame and the timestamp of each word is calculated. For example, if the timestamp of a speech frame is 0.4-0.6 seconds and the timestamp of a word is 0.3-0.8 seconds, the overlap time is 0.4-0.6 seconds, and the overlap ratio is (0.6-0.4) / (0.8-0.3) = 40%. If the overlap ratio exceeds a preset threshold, such as 30%, a preliminary correspondence between the speech frame feature and the semantic feature of the word is established. For the speech rate sliding window feature, the window timestamp and the sentence timestamp are calculated. The coverage range is defined as follows: for example, if a speech rate window is 1-2 seconds and a sentence timestamp is 1.2-1.8 seconds, the coverage duration is 0.6 seconds, accounting for 60% of the window duration. If the coverage ratio exceeds 50%, a correspondence is established between the speech rate feature of the window and the semantic feature of the sentence. For pause features, it is determined whether the pause timestamp is within the time interval of a word or sentence. For example, if the pause is 1.5-1.7 seconds and a sentence is 1.3-1.9 seconds, then the pause is within that sentence, and an association is established between the pause feature and the corresponding word or sentence semantic feature. Through the above methods, a fine-grained temporal correspondence table between the two modal feature sequences is finally formed, determining the semantic sentiment feature element corresponding to each speech sentiment feature element and the degree of association.

[0071] Step 221: Based on fine-grained temporal correspondence, a cross-modal feature alignment framework is constructed to organize and map features of two modalities on a shared temporal dimension. This includes: calculating the dynamic alignment path between each frame in the speech feature sequence and each word unit in the text feature sequence based on temporal correction coefficients, establishing a temporal mapping relationship from speech to text; mapping the feature vector of each frame in the speech feature sequence to the timestamp position of the corresponding word unit in the text feature sequence according to the dynamic alignment path, realizing the projection and alignment of cross-modal features under a unified temporal coordinate; and, under the unified temporal coordinate, mapping the mapped speech features and the corresponding text word unit features. Alignment and splicing are performed one by one to reassemble a synchronous multimodal fusion feature sequence. Specifically, this includes: based on the temporal correspondence established in step 220, statistically analyzing the timestamp distribution of all speech emotion features and semantic emotion features, determining a shared time axis covering the entire call duration, dividing the shared time axis into several aligned time units at fixed time intervals, such as 0.1 seconds per interval, with the interval length determined based on the average duration of speech frames and text words, ensuring that each interval contains at least one speech frame and one word. Each unit corresponds to a specific time interval, such as the first unit 0-0.1 seconds, the second unit 0.1-0.2 seconds, until the call end time is covered.

[0072] For speech emotion features, the overlap between each feature element, such as the timestamp of the intonation frame, speech rate window, intensity frame, and pause segment, and each aligned time unit is determined. If the overlap time accounts for more than 40% of the total duration of the feature element, the value of that feature element, such as fundamental frequency, speech rate, energy, or pause duration, is assigned to the corresponding aligned time unit. If a feature element covers multiple aligned time units, the value is split according to the overlap duration ratio. For example, if a speech rate window of 1-2 seconds covers 10 0.1-second units, each unit is assigned 10% of the speech rate value. For semantic emotion features, the overlap relationship between the timestamp of word or sentence-level features and the aligned time units is also determined, and the emotion tendency score, emotion word intensity, etc., are assigned to the corresponding units according to the overlap ratio. If there is no directly corresponding semantic feature element in an aligned time unit, linear interpolation is performed using the semantic feature values ​​of adjacent units. For example, if unit 5 has no semantic feature, unit 4 scores 0.2, unit 6 scores 0.4, and the interval is 0.1 seconds, then unit 5 scores 0.2. + (0.4-0.2)×(5-4) / (6-4)=0.3, ensuring that each aligned time unit has a semantic sentiment feature value.

[0073] Using aligned time units as rows and speech emotion feature types (intonation, speech rate, intensity, pauses) and semantic emotion feature types (emotional tendency, emotional word intensity, keyword co-occurrence frequency) as columns, a two-dimensional feature alignment matrix is ​​constructed. The value of each cell in the matrix is ​​the allocation or calculated value of that type of feature within the corresponding aligned time unit, realizing the unified organization of features of the two modalities in the shared temporal dimension and aligned time unit sequence. At the same time, the source of each feature value is marked in the matrix, such as speech features from a certain frame / window, semantic features from a certain word / sentence, and the degree of association and overlap ratio, forming a complete cross-modal feature alignment framework that supports the calculation of association between features.

[0074] Step 222: Based on the cross-modal feature alignment framework, calculate the dynamic similarity measure and correlation coefficient between speech emotion features and semantic emotion features within the same time interval. Specifically, this includes: using alignment time units based on the cross-modal feature alignment framework, with a fixed length, such as 1 second, the length is determined according to the average sentence duration of the call, to ensure that each interval contains multiple alignment units to divide the analysis time interval. Each interval consists of consecutive alignment time units, such as the first interval containing units 1-10, corresponding to 0-1 seconds; the second interval containing units 11-20, corresponding to 1-2 seconds. At the same time, the interval division can also be adjusted according to the sentence boundaries of the semantic text to ensure that each analysis time interval contains complete sentence semantics. For example, if a sentence corresponds to 1.2-1.8 seconds, then the analysis interval is adjusted to 1-2 seconds to cover that sentence.

[0075] For each analysis time interval, the speech sentiment features of all aligned units within that interval are extracted, such as the mean fundamental frequency of pitch, the mean energy of sound intensity, and semantic sentiment features, such as sentiment tendency score and the sum of sentiment word intensities. Taking the similarity calculation between the mean fundamental frequency of pitch and the sentiment tendency score as an example, firstly, the sequence change trend of the mean fundamental frequency within that interval is calculated, and the slope is obtained through linear fitting. A positive slope indicates an increase in the fundamental frequency, while a negative slope indicates a decrease. The sequence change trend of the sentiment tendency score is also calculated, and the fitting slope is calculated similarly. Then, the consistency of the two trends is calculated. If both are positive or both are negative, the consistency is recorded as 1; otherwise, it is recorded as 0. Combining the similarity of the fluctuation amplitude of the two values, the fundamental frequency is calculated. The standard deviation of the mean and the standard deviation of the sentiment score are used to calculate the similarity of tone and sentiment within each analysis interval. For example, the ratio of the standard deviation of the fundamental frequency to the standard deviation of the sentiment score is calculated. If the ratio is within the range of 0.8-1.2, the fluctuation similarity is recorded as 0.8-1.0. If the ratio is outside the range, it is reduced proportionally. The trend consistency and fluctuation similarity are weighted and summed, with the weights set according to the importance of the features. For example, the trend consistency weight is 0.6 and the fluctuation similarity weight is 0.4. This yields the dynamic similarity measure of tone and sentiment within the interval, with a value of 0-1. The closer to 1, the higher the similarity. Similarly, the dynamic similarity measure of speech rate, intensity, pauses, and each semantic sentiment feature within each analysis interval is calculated using the same method.

[0076] For each analysis time interval, the speech emotion feature value sequence within that interval, such as the intensity energy value of each aligned unit, and the corresponding semantic emotion feature value sequence, such as the emotion word intensity value of each aligned unit, are used as two sets of data. The linear correlation coefficient between the two is calculated. Specifically, the mean of the two sets of data is first calculated, such as the mean intensity = sum of all unit intensity values ​​ / number of units, and the mean emotion word intensity = sum of all unit intensity values ​​ / number of units. Then, the deviation of each element in each set of data from the mean of that set is calculated, such as the deviation of a unit's intensity = unit's intensity value - mean intensity, and the deviation of emotion intensity = unit's intensity value - mean intensity value. Unit intensity value - mean intensity value; then calculate the sum of the products of the two sets of deviations (the sum of the intensity deviations of all units × the emotional intensity deviations), and the sum of the squares of the two sets of deviations (the sum of the squares of the intensity deviations of all units, the sum of the squares of the emotional intensity deviations of all units); finally, divide the sum of the deviation products by the square root of the product of the sums of the squares of the two sets of deviations to obtain the linear correlation coefficient, which takes values ​​of -1 to -1, where 1 indicates a perfect positive correlation, -1 indicates a perfect negative correlation, and 0 indicates no correlation. For each combination of speech emotional features and semantic emotional features, the correlation coefficient for each analysis time interval is calculated using this process.

[0077] Step 223: Based on the dynamic similarity measure and correlation coefficient, generate a set of temporal correction coefficients for feature alignment. Specifically, this includes assigning different weights to the dynamic similarity measure and correlation coefficient according to the importance of the feature type. For example, for the association between tone and sentiment, the dynamic similarity measure reflects the matching degree of trend and fluctuation, and has a greater impact on feature alignment, so it is assigned a weight of 0.6; the correlation coefficient reflects the degree of linear association, so it is assigned a weight of 0.4; for the association between pauses and semantic logical coherence, the correlation coefficient better reflects the matching between pause position and semantic structure, so it is assigned a weight of 0.6, and the dynamic similarity measure is assigned a weight of 0.4. The weight allocation needs to be combined with the emotional expression rules in the call center scenario. For example, changes in voice tone are usually more synchronized with semantic sentiment. The weight is determined by statistically analyzing the correlation between the two coefficients and the final sentiment judgment accuracy in the call data. For example, the higher the correlation coefficient between a certain coefficient and the accuracy, the greater its weight.

[0078] For each analysis time interval, the dynamic similarity measure of a certain cross-modal feature group within that interval, such as speech rate and sentiment tendency, is multiplied by its corresponding weight, and then the correlation coefficient of that feature group is multiplied by its corresponding weight to obtain the preliminary correction coefficient of that feature group within that interval. For example, similarity measure 0.8×0.5 + correlation coefficient 0.7×0.5 = 0.75. Then, the preliminary correction coefficients of all cross-modal feature groups (tone-sentiment tendency, speech rate-sentiment tendency, intensity-sentiment word intensity, pause-semantic coherence, etc.) within that interval are calculated, and their average value is calculated. For example, if the preliminary coefficients of the four groups are 0.75, 0.8, 0.7, and 0.85 respectively, the average value = (0.75+0.8+0.7+0.85) / 4 = 0.775. This average value is used as the interval time-series correction coefficient for that analysis time interval, reflecting the overall alignment degree of the two modal features within that interval. The closer the coefficient is to 1, the higher the alignment degree.

[0079] After obtaining the interval time series correction coefficients for all analysis time intervals, and considering the duration percentage of each interval (e.g., if an interval is 1 second long and the total call duration is 10 seconds, representing 10% of the total call duration), calculate the global weighted average correction coefficient. If the coefficients for the three intervals are 0.7, 0.8, and 0.9, and their duration percentages are 20%, 30%, and 50%, respectively, the global coefficient = 0.7 × 20% + 0.8 × 30% + 0.9 × 50% = 0.83. Simultaneously, to reflect dynamic changes in the time series, the correction coefficients are calculated according to the analysis time intervals. The correction coefficients for each interval are matched with the starting timestamp of that interval to form a time-series correction coefficient sequence. For example, the coefficient for time 0-1 seconds is 0.7, the coefficient for 1-2 seconds is 0.8, and the coefficient for 2-3 seconds is 0.9. In addition, outliers in the coefficient sequence, such as the coefficient of a certain interval being much lower than that of other intervals with a difference exceeding 0.3, are corrected by linear interpolation of the coefficients of adjacent intervals. For example, the coefficient of interval 2 is 0.2, the coefficient of interval 1 is 0.7, and the coefficient of interval 3 is 0.8. After correction, the coefficient of interval 2 is 0.7 + (0.8-0.7)×(2-1) / (3-1)=0.75. Finally, a set of time-series correction coefficients that includes the global average level and reflects the dynamic changes of each time period is generated.

[0080] By combining precise timestamps of speech frames and text words, a fine-grained temporal correspondence is established, enabling each speech sentiment feature element to match the most relevant semantic sentiment feature element. For speech features at different temporal granularities, such as intonation, speech rate, intensity, and pauses, correspondences are established with word-level and sentence-level semantic features, respectively. This covers multi-dimensional association scenarios from micro to macro levels, meeting diverse sentiment analysis needs. By constructing a shared temporal dimension, previously independent speech and semantic features are integrated into the same framework, resolving the difficulty of direct comparison and analysis between the two modal features due to differences in temporal granularity. This provides a unified temporal reference standard for cross-modal features. Through numerical decomposition and linear interpolation, each aligned temporal unit is ensured. Each feature has corresponding speech and semantic feature values, avoiding analytical gaps caused by the lack of feature data in certain time intervals. This is especially true for silent speech segments or text intervals without explicit sentiment words. By interpolating to supplement semantic features, the entire call process is fully covered, improving the comprehensiveness of sentiment analysis. The dynamic similarity measure not only focuses on the correlation of feature values ​​but also considers the changing trends and fluctuations of feature sequences. It can capture the synchronicity of cross-modal features changing over time. Compared with single numerical comparisons, it can better reflect the real emotional association patterns and provide a more realistic basis for feature alignment. The generated temporal correction coefficient can accurately reflect the degree of alignment of cross-modal features in different time intervals, avoiding correction bias caused by subjective experience judgment.

[0081] In a preferred embodiment of the present invention, step 3 above, which utilizes a time-series correction coefficient to perform multimodal synchronous processing on speech emotion features and semantic emotion features, identifies and corrects time-series misalignments and semantic inconsistencies caused by slips of the tongue, pauses, and transcription errors, and generates a consistent multimodal feature sequence, may include:

[0082] Step 330 involves applying the temporal correction coefficients to the speech emotion feature sequence and the semantic emotion feature sequence respectively, establishing a mapping relationship between the two modal features. Specifically, this includes: first, obtaining the generated temporal correction coefficient sequence, including the correction coefficient and corresponding timestamp for each analysis time interval; then, splitting the speech emotion feature sequence into intervals corresponding to the correction coefficient sequence according to the timestamps. For example, if the correction coefficient sequence has a coefficient of 0.8 for 0-1 seconds and a coefficient of 0.75 for 1-2 seconds, the feature segments for 0-1 seconds and 1-2 seconds in the speech emotion feature sequence are separated. The speech emotion feature values ​​within each interval, such as the fundamental frequency of pitch and the energy value of sound intensity, are adjusted using coefficients. If the correction coefficient is greater than 0.9, it indicates high alignment between the two modal features in that interval, requiring no significant adjustment, and the feature values ​​remain unchanged. If the correction coefficient is between 0.7 and 0.9, indicating moderate alignment, the feature value is multiplied by the ratio of the correction coefficient to 0.9, such as a coefficient of 0.8 and a ratio of 0.8. / 0.9≈0.89, feature value ×0.89, fine-tune feature value to closely match semantic feature trend; if the correction coefficient is less than 0.7, the alignment is low, then first calculate the difference between the speech feature value of this interval and the adjacent high alignment interval, the coefficient ≥0.8 speech feature value, and then use the correction coefficient to weight the difference. For example, if the difference is 5, the coefficient is 0.6, the weighted difference = 5 × 0.6 = 3. Subtract the weighted difference from the current interval feature value. If the current value is higher than the adjacent interval, or the weighted difference is added, if the current value is lower than the adjacent interval, the feature value is adaptively adjusted.

[0083] Using the same interval division method as for speech emotion features, the semantic emotion feature sequence is split into corresponding interval segments. For the semantic feature values ​​of each interval segment, such as emotion tendency score and emotion word intensity, adjustments are made according to a correction coefficient. If the correction coefficient is ≥0.8, the alignment is high, and only core emotion words in the semantic features, such as anger and satisfaction with an intensity ≥0.7, are retained, while meaningless filler words, such as "um" and "this," are removed. If the correction coefficient is between 0.6 and 0.8, the alignment is moderate, and in addition to removing filler words, the emotion tendency score is corrected, and the previous interval is calculated. The mean sentiment score of the next adjacent interval is calculated by multiplying the difference between the current score and the mean by a correction coefficient, and then adding it to the current score. For example, if the current score is 0.3, the mean is 0.5, and the correction coefficient is 0.7, the corrected score is 0.3 + (0.5 - 0.3) × 0.7 = 0.44. If the correction coefficient is less than 0.6, indicating low alignment, the linear interpolation result of the semantic features of the preceding and following intervals is used to replace the feature value of the current interval. For example, if the score of the preceding interval is 0.2, the score of the following interval is 0.6, and the current interval is in the middle, the interpolation score is (0.2 + 0.6) / 2 = 0.4.

[0084] After adjusting the feature values ​​of each interval, the timestamps of the corrected speech emotion feature sequence are used as the benchmark. The timestamps of the corrected semantic emotion feature sequence are matched one by one. The time overlap ratio of each speech feature element, such as the tone of a certain frame and the semantic feature element, such as the emotion of a certain word, is calculated. The calculation method is the same as in step 220. If the overlap ratio is ≥30%, a one-to-one or one-to-many mapping relationship is established. For example, one long speech frame corresponds to two short word semantic features, or two short speech frames correspond to one long word semantic feature. The feature values, timestamps and correction coefficients of both mapping parties are recorded to form a complete mapping relationship table and clarify the corresponding relationship between the two modal features.

[0085] Step 331: Based on the mapping relationship, the final alignment path is determined by calculating the cumulative distance between the two modal feature sequences, and the time scale of the speech emotion feature sequence and the semantic emotion feature sequence is adjusted to achieve temporal alignment of the feature sequences. Specifically, this includes setting the corrected speech emotion feature sequence as a sequence. ,Include One element, , … The semantic sentiment feature sequence is set as a sequence. ,Include One element, , … , in sequence The elements are rows and sequences. Using the elements as columns, construct a distance matrix, where each cell in the matrix ( , The distance value is calculated by... and To determine the feature difference, for numerical features such as pitch fundamental frequency and sentiment tendency score, calculate the absolute difference between the two and divide it by the maximum value of the two feature values. 200Hz base frequency The corresponding semantic feature score is 0.5, with a maximum value of 200. The distance is |200-0.5| / 200≈0.9975. Here, the feature values ​​need to be normalized first, mapping the semantic feature scores of 0-1 to the same numerical range of the speech features, such as a fundamental frequency of 80-300Hz. A score of 0 corresponds to 80Hz, 1 corresponds to 300Hz, and 0.5 corresponds to 190Hz. Then, the absolute difference is calculated as |200-190| / (300-80)≈0.045. For non-numerical features, such as pause type and keyword type, if the types are consistent, the distance is 0; if they are inconsistent, the distance is 1. Then, starting from the top left corner of the matrix ( , Start by calculating the cumulative distance in the three directions: right, down, and lower right. Move to the right ( , The cumulative distance of ) = current cumulative distance + ( , The distance value; move down ( , The cumulative distance of ) = current cumulative distance + ( , The distance value; move to the lower right ( , The cumulative distance of ) = current cumulative distance + ( , The distance value is multiplied by 1.2, giving higher weight to the diagonal movement, prioritizing the selection of time-synchronized elements, until the calculation reaches the bottom right corner of the matrix ( , ), which gives the cumulative distance of all possible paths.

[0086] From the bottom right corner of the matrix ( , Backtracking in reverse, comparing the cumulative distances to the left, up, and upper left for each position, the direction with the smallest cumulative distance is selected as the backtracking path. For example, if the current position has a cumulative distance of 5, the cumulative distances to the left are 4.8, up are 4.9, and upper left are 4.5, then the path is upper left, and so on until the backtracking reaches the upper left corner of the matrix. , ), forming a line from ( , )arrive( , The path with the smallest cumulative distance is the final alignment path, and each cell on the path ( , )correspond and The final alignment relationship; based on the final alignment path, the statistics of each speech feature element. Corresponding semantic feature elements Timestamp range, calculation Timestamp and timestamp deviation, such as Timestamp 0.5-0.7 seconds, The timestamp is 0.6-0.8 seconds, with a deviation of 0.1 seconds. The time scale of the speech emotion feature sequence is adjusted. If the deviation is positive, indicating that the speech features lag behind the semantic features, the timestamp of that speech feature element is shifted forward by the deviation value. Adjust to 0.4-0.6 seconds; if the deviation is negative, the speech feature leads the semantic feature, then shift the deviation value backward; if one speech feature element corresponds to multiple semantic feature elements, such as correspond , Then calculate the average timestamp of these semantic feature elements, and... The timestamp is adjusted to overlap with the average timestamp; similarly, for multiple speech feature elements corresponding to a semantic feature sequence... Calculate the average timestamp of the speech feature elements and adjust The timestamp overlaps with it. After the adjustment is completed, the time overlap ratio of all aligned elements is re-verified to ensure that it is ≥50%, thus achieving temporal alignment of the two modal feature sequences.

[0087] Step 332: On the basis of temporal alignment, detect and correct semantic inconsistency problems caused by slips of the tongue, repetitions, and automatic speech recognition errors in the speech signal. Specifically, it includes: Based on the two-modal feature sequences after temporal alignment, detection is carried out from three dimensions. One is slip-of-the-tongue detection. Compare the pause features in the speech emotion features, such as the short pause frequency, with the semantic text. If the short pause frequency in a certain segment of speech is ≥ 3 times per second, far exceeding the normal speaking pause frequency of 1 time per second, and there are repeated words in the corresponding text, such as "I I I need a refund" or meaningless words, such as "uh refund", it is determined that there is a slip of the tongue. Second, detection of semantic breaks caused by pauses. Check the long pauses in the pause features, the text positions corresponding to pauses ≥ 1 second. If the long pause is in the middle of the sentence, without punctuation marks such as commas and full stops, and the text semantics before and after the pause are not related, such as "I want to consult the logistics... Your service is too bad", it is determined that there is a semantic break. Third, transcription error detection. Compare the transcribed text with the intensity and intonation features in the speech emotion features. If the average value of the intensity energy of a certain segment of speech is ≥ 1.5 times the overall average value, indicating that the speaker emphasizes this part, but there are no emotion words or keywords in the corresponding text, such as actually saying "this problem is very serious" and being transcribed as "this problem is very", or the keywords in the text do not match the speech field, such as in financial customer service, the speech corresponds to "repayment" and is transcribed as "return of funds", it is determined that there is a transcription error.

[0088] For different problem types, corresponding correction methods are adopted. In terms of slip-of-the-tongue correction, for the repeated words in the text, retain the first-occurring word. For example, "I I I need a refund" is corrected to "I need a refund", and for meaningless words such as "uh" and "um", they are directly deleted. At the same time, refer to the speech intonation features. If the average value of the fundamental frequency of the slip-of-the-tongue segment is higher than that of the normal segment by ≥ 1.2 times, it is judged that the slip of the tongue is caused by nervousness, and after correcting the text, add a note that the speaker is slightly nervous. In terms of correcting semantic breaks caused by pauses, check the speech intonation trends before and after the long pause. If the intonation rises before the pause and falls after the pause, it is judged as a thought conversion, and add a comma at the text pause position, such as "I want to consult the logistics, your service is too bad". If the intonation before and after the pause is both low ≤ 0.8 times the normal average value, it is judged that the pause is caused by a low mood, and add a short silence in the text, with a low mood. In terms of correcting transcription errors, combine the acoustic features of the speech signal, such as the acoustic feature differences between "repayment" and "return of funds", and call the domain keyword dictionary. For example, the financial customer service dictionary contains "repayment" and no "return of funds", and replace the wrong word with the correct word that matches the acoustic feature in the dictionary, such as correcting "return of funds" to "repayment". If the text in the segment with strong emphasis on intensity is missing, refer to the semantics of the context. For example, the previous sentence is "logistics delay" and the next sentence is "affect my use", and supplement the missing content. For example, "this problem is very serious" is corrected, and then 10% - 20% of the text is randomly inspected by humans to ensure that the correction accuracy rate ≥ 95%.

[0089] Step 333 involves fusing the corrected speech emotion features and semantic emotion features to generate a temporally consistent and coordinated multimodal feature sequence. Specifically, this includes determining the weights based on the reliability of the two modal features. Reliability is evaluated using the temporal correction coefficient from step 223. If the correction coefficient for a certain interval is ≥0.8, the alignment is high and the feature reliability is strong, then the weights of both the speech emotion features and semantic emotion features are set to 0.5. If the correction coefficient is between 0.6 and 0.8, one side has slightly higher reliability. The consistency between the speech features and semantic features within that interval is then compared, such as whether the intonation trend and the emotion tendency trend are consistent. If the weights remain at 0.5, and there is inconsistency, the weight of the feature with higher reliability is set to 0.6, and the other is set to 0.4. For example, if the consistency rate between tone trend and sentiment trend is <60%, and the dynamic similarity measure of voice features is higher, then the voice weight is 0.6 and the semantic weight is 0.4. If the correction coefficient is <0.6, and one side has low reliability, then the weight of the feature with higher reliability is set to 0.7, and the other is set to 0.3. At the same time, combined with the needs of call center scenarios, such as customer service quality monitoring focusing more on semantic content, the basic weight of semantic sentiment features is increased by 0.1, such as adjusting the original 0.5 to 0.6 and 0.4 to 0.5.

[0090] For the corrected numerical speech features, such as the mean fundamental frequency of intonation and speech rate, and semantic features, such as sentiment score and sentiment word intensity, a weighted sum is performed according to a determined weight. For example, the mean fundamental frequency of speech intonation in a certain interval is 0.7 after normalization, mapped to the range of 0-1, with a weight of 0.4; the semantic sentiment score is 0.6, with a weight of 0.6, and the fused feature value = 0.7 × 0.4 + 0.6 × 0.6 = 0.64; the speech rate is 2 words / second, normalized to 0.5, with a weight of 0.3; the semantic indicators reflecting the fluency of speech, such as sentence completeness, are 0.8, with a weight of 0.7, and the fused fluency feature value = 0.5 × 0.3 + 0.8 × 0.7 = 0.71. During the fusion process, if the difference between two feature values ​​is ≥ 0.3, the temporal correction coefficient is re-verified after normalization. If the coefficient is ≥ 0.7, the average of the two is taken as the fused value; if the coefficient is < 0.7, the feature value with the higher weight is used.

[0091] For non-numerical features, such as pause type and sentiment labels, a method of prioritizing main features and supplementary descriptions is used for fusion. If the speech feature is labeled as a long pause (thinking type), the semantic feature is labeled as a sentence transition, and the fused feature is labeled as a long pause (thinking type, accompanied by sentence transition). If the speech feature detects emotional excitement (high pitch), the semantic feature is labeled as negative emotion (including angry words), and the fused feature is labeled as emotional excitement (negative, angry). All fused numerical features are arranged in timestamp order, and non-numerical feature labels are mapped to the corresponding time intervals, forming a multimodal feature sequence containing timestamps, fused numerical features, and fused labels. Finally, the temporal consistency of all features in the sequence is verified, with an overlap ratio of ≥50% and semantic consistency. The labels are matched with the trend of numerical features, such as emotional excitement corresponding to a fused sentiment value of ≥0.7, to ensure the sequence is consistent.

[0092] By using temporal correction coefficients to specifically adjust the feature sequences of the two modalities, misalignment caused by deviations in the original feature values ​​is avoided, making the established mapping relationship more consistent with the emotional expression logic in actual conversations. Differentiated adjustment strategies are adopted for different alignment intervals, which not only preserves the original information of high-reliability features but also corrects the deviations of low-reliability features, making the two modal features more consistent in terms of numerical range and trend. The final alignment path is selected by cumulative distance calculation, avoiding the subjectivity of manually specifying the path. Combined with time scale adjustment, the problem of temporal misalignment caused by differences in the acquisition accuracy of the two modal features is solved, so that each speech feature element can be synchronized with the most matching semantic feature element in time, improving the temporal accuracy of sentiment analysis. By detecting and accurately locating slips of the tongue and transcription errors through multi-dimensional detection, and then making targeted corrections, semantic deviations that are difficult to detect by relying solely on text are solved, making the transcribed text more consistent with the speaker's true intentions and providing an accurate textual basis for semantic sentiment analysis. Emotion annotations are added during the correction process to enrich the contextual information of semantic features and avoid sentiment misjudgment caused by relying solely on textual units.

[0093] In a preferred embodiment of the present invention, step 4 above, based on the multimodal feature sequence, employs a cross-modal attention mechanism to achieve deep fusion of speech emotion features and semantic emotion features, and dynamically adjusts the influence of different modalities in emotion analysis according to the confidence and semantic consistency of each modality, generating a joint representation vector including global sentiment tendency, may include:

[0094] Step 440 involves performing cross-modal temporal alignment on the speech emotion feature sequence and the text emotion feature sequence to obtain the aligned multimodal feature sequence. Specifically, this includes: first, obtaining the speech emotion feature sequence corrected in step 333, containing numerical features of intonation, speech rate, intensity, and pauses, along with corresponding timestamps; and second, obtaining the text emotion feature sequence, containing emotion tendency score, emotion word intensity, keyword co-occurrence frequency, and corresponding timestamps. For the speech feature sequence, all numerical features are normalized to the 0-1 range. For example, the average fundamental frequency of intonation, originally 80-300Hz, is converted to a normalized value of 0-1 using the formula (fundamental frequency value - 80) / (300 - 80); the speech rate, originally 1-5 words / second, is normalized using (speech rate value - 1) / (5 - 1); the intensity energy value and pause duration are processed in a similar manner to ensure a uniform numerical range for each speech feature dimension. For the text feature sequence, the emotion tendency score (originally - 1 - 1) is also normalized using (score + 1) / 2. The values ​​are converted to normalized values ​​of 0-1. The intensity of sentiment words and the co-occurrence frequency of keywords are also normalized according to the difference between their respective maximum and minimum values, eliminating the impact of numerical differences in different feature dimensions on alignment.

[0095] Based on the timestamps of the speech emotion feature sequence, it is divided into multiple aligned segments at fixed time intervals, such as 0.1 seconds per interval, consistent with the alignment time unit in step 221. Each segment contains the normalized values ​​of all speech features within that time interval. For the text emotion feature sequence, aligned segments are also divided at 0.1-second time intervals. If there are no text features within a certain time interval, such as a speech silence segment, linear interpolation is performed to supplement it using the text feature values ​​of the preceding and following segments. For example, if the normalized value of the emotion tendency of the preceding segment is 0.3, the following segment is 0.5, and the value of the blank segment in the middle is 0.3 + (0.5 - 0.3) × 0.5. =0.4, then calculate the time matching degree between each speech-aligned segment and the corresponding time interval text-aligned segment, count the overlap duration of the speech feature timestamp and the text feature timestamp within the segment, divide by the total segment duration (0.1 seconds) to obtain the time matching degree. For example, if the overlap is 0.08 seconds, the matching degree = 0.08 / 0.1 = 0.8. If the matching degree is ≥0.6, directly pair the speech segment with the text segment; if the matching degree is <0.6, adjust the time interval of the text segment, such as shifting it forward or backward by 0.05 seconds, and recalculate the matching degree until a text segment with a matching degree ≥0.6 is found, forming a one-to-one corresponding alignment segment pair.

[0096] All aligned segment pairs are arranged in chronological order. Each segment pair contains a normalized vector of speech features within that time interval, such as intonation, speech rate, intensity, and pauses, each occupying one dimension, forming a 4-dimensional vector. A normalized vector of text features, such as sentiment tendency, sentiment word intensity, and keyword co-occurrence frequency, each occupying one dimension, forms a 3-dimensional vector. The temporal matching degree of each segment pair is recorded. The feature vectors of all segment pairs are validated, and the cosine similarity between the speech vector and the text vector is calculated by dividing the vector dot product by the product of the two vector magnitudes. If the similarity is <0.3 (indicating a large difference in feature trends), the temporal alignment is rechecked for accuracy. If necessary, step 2 is repeated to adjust the aligned segments, ensuring a cosine similarity ≥0.3. Finally, the validated segment pairs are combined in chronological order to form the aligned multimodal feature sequence.

[0097] Step 441: Input the aligned multimodal feature sequence into the cross-modal attention mechanism. Perform bidirectional attention calculations using speech features as queries and text features as keys, and vice versa, to achieve intermodal interaction modeling and extract deep semantic association features to obtain cross-modal fusion features. Specifically, this includes: organizing the speech feature vectors and text feature vectors in the aligned multimodal feature sequence into matrix form, and the speech feature matrix... , dimension × , For the number of fragment pairs, For speech features, such as 4D, text feature matrix , dimension × , To enhance feature representation, the text feature dimension can be increased to 3D. and Perform linear transformations respectively, and... Multiply by a learnable weight matrix Dimension × , Assuming the attention mechanism has key dimensions, such as 5 dimensions, we obtain the key vector matrix of speech features. and query vector matrix , here = To simplify calculations; Multiply by the weight matrix Dimension × The key vector matrix of text features is obtained. Sum value vector matrix , = .

[0098] Attention computation, which uses speech as the query and text as the key, first calculates the query vector matrix. With key vector matrix The similarity, for Each query vector in Dimension 1× ,and Each key vector in Dimension 1× Calculate the dot product The similarity matrix is ​​obtained. Dimension × To avoid excessively large values, the similarity matrix... Divide each element in by ,like =5, ≈2.236, yielding the normalized similarity matrix. Then to Softmax is calculated for each row, and the exponent of each row element is taken first. ^ , (The element in that row is used as the index), and then the sum of the indices of all elements in that row is used to obtain the attention weight matrix. Dimension × The weight values ​​range from 0 to 1, with a total weight of 1 for each row, reflecting the degree of attention each speech segment pays to different text segments. Finally, the attention weight matrix is... With text value vector matrix Perform matrix multiplication × The text attention feature matrix guided by voice is obtained. Dimension × This matrix integrates information from text features that is closely related to speech features.

[0099] Attention calculation, using text as the query and voice as the key, employs a symmetrical approach to the above, processing the text feature matrix... query vector matrix , = With speech feature matrix Key vector matrix Calculate the similarity to obtain the similarity matrix. ', dimension × Divide by the same amount Then, softmax calculation is performed to obtain the attention weight matrix. Dimension × This reflects the degree of attention each text segment pays to different speech segments. With speech value vector matrix , = Perform matrix multiplication to obtain the text-guided speech attention feature matrix. Dimension × × , simplified here × .

[0100] Voice-guided text attention feature matrix Text-guided speech attention feature matrix Adding elements together yields a preliminary fusion matrix. Dimension × To highlight the deep semantic connections, A nonlinear transformation is performed, mapping each element to the 0-1 range using the sigmoid function (f(x)=1 / (1+e^(-x))) to enhance the discriminative power of the features. Simultaneously, the original speech feature matrix is ​​transformed... and text feature matrix Mapped to respectively through linear transformation Dimension, to obtain '( × )and '( × ),and Added according to weight ( '×0.2+ '×0.2+ (×0.6), finally obtaining the cross-modal fusion feature matrix. Dimension × Each row vector corresponds to a deep semantic association feature of a time segment.

[0101] Step 442: Based on cross-modal fusion features, calculate the acoustic confidence index of speech features and the semantic consistency index of text features respectively. Dynamically determine the importance of speech modality and text modality in emotion recognition based on these two types of indices. Specifically, this includes: acoustic confidence is based on the stability and discriminativeness assessment of speech features, divided into two sub-indices, one of which is the feature stability index. For cross-modal fusion feature matrix The features derived from speech, namely After mapping, the cosine similarity between the feature vector of each time segment and the feature vectors of its five adjacent segments is calculated. The average value is taken as the local stability of that segment. Then, the mean of the local stability of all segments is calculated to obtain the result. The first is a range of 0-1, where a higher value indicates more stable speech features; the second is an emotion discrimination index. The statistical analysis focuses on dimensions of speech features strongly correlated with emotion, such as the standard deviation of pitch fundamental frequency and the maximum value of sound intensity energy, showing differences in values ​​under different emotion labels, such as positive, negative, and neutral. The greater the difference, the better. The higher the acoustic confidence level, the more accurate it is to calculate the variance of the feature values ​​under different labels and then normalize it to the 0-1 range. = ×0.6+ ×0.4, the weight is set according to the degree of influence of speech features on emotion. The closer the value is to 1, the higher the confidence level of the speech modality.

[0102] Semantic consistency is evaluated based on the matching degree between text features and context and sentiment logic. It also includes two sub-indicators: one is the context consistency index. For a text sentiment feature sequence, the Pearson correlation coefficient between the sentiment score of each statement and the scores of the two statements before and after it is calculated. This is achieved by calculating the covariance divided by the product of the standard deviations of the two sequences, taking the mean of all correlation coefficients, and normalizing them to the range of 0-1. The higher the value, the more coherent the contextual semantics; secondly, the sentiment-logic consistency index. Based on the domain sentiment lexicon, such as the mapping relationship between common sentiment words in call centers and their corresponding sentiment tendencies, it is determined whether the sentiment tendencies of the sentiment words in the text are consistent with the overall sentiment score. For example, satisfaction corresponds to a positive tendency. If the text contains satisfaction but the score is negative, then they are inconsistent. The percentage of consistent sentiment words is then calculated as a measure of their consistency. The semantic consistency index is in the range of 0-1. = ×0.5+ ×0.5, The closer to 1, the higher the text modality consistency.

[0103] Calculate the relative weights of the two modal indices, if ≥ Speech modal basic weights = / ( + Text modality basic weights =1- ;like > , = / ( + ), =1- Then, the weights are adjusted based on the alignment effect of the cross-modal fusion features. If the cosine similarity between the index of one modality and the cross-modal fusion features is higher than that of another modality, such as... and The similarity is 0.8. and If the similarity is 0.6, then the weight of this modality increases by 0.1, and the weight of the other modality decreases by 0.1, ensuring that the total weight is 1. For example, =0.8, =0.7, base weight =0.8 / (0.8+0.7)=0.53, =0.47; if and The similarity is higher after adjustment. =0.63, =0.37, thus obtaining the importance of speech modality. Importance of text modality The sum of the two is 1.

[0104] Step 443: Based on modal importance, adaptively fuse cross-modal fusion features to generate a joint sentiment representation vector that includes global contextual sentiment semantic information. Specifically, this includes: based on the modal importance determined in step 442... and For cross-modal fusion feature matrix Perform weighted splitting, The features derived from speech, Related content multiplied Features derived from the text Related content multiplied The weighted speech fusion features are obtained. = × Weighted text fusion features = × Meanwhile, the original speech feature matrix and text feature matrix Perform global pooling processing separately and calculate... The global speech feature vector is obtained by averaging the feature vectors of all time segments. Dimension 1× Similarly, calculate global feature vector of text Dimension 1× And linearly transform both to Dimension, to obtain (1×) )and (1×) ).

[0105] Weighted and By adding elements together, we obtain the time-dimensional fused features. Dimension × ;right Global average pooling is performed to calculate the average of the feature vectors across all time segments, resulting in the global temporal fusion feature. Dimension 1× Then and , To perform fusion, first calculate the maximum value of the three elements to obtain the maximum fusion feature. , 1× Then calculate the average elemental value of the three to obtain the average fusion characteristic. , 1× Finally, and By concatenating them, a concatenated feature vector is formed. Dimension 1×2 This vector contains multi-scale feature information in both the time and global dimensions.

[0106] splicing feature vectors Dimensionality reduction and optimization were performed by retaining principal components with a variance contribution of ≥90% through principal component analysis (PCA), reducing the dimensionality from 2... Down to ,like =5,2 =10, decreased to =8, removing redundant information, then normalizing the dimensionality-reduced vector by dividing each element by the vector norm and the square root of the sum of squares of all elements to ensure the vector magnitude is 1, enhancing the stability of subsequent sentiment analysis. Finally, global sentiment information is appended to the end of the vector, and the cross-modal fusion feature matrix is ​​calculated. The sentiment score for all time segments is calculated by multiplying the feature vector with a preset sentiment vector, and the average value is taken as the global sentiment value, ranging from 0 to 1, where 0 corresponds to negative and 1 corresponds to positive. This value is added to the normalized vector to form the final joint sentiment representation vector, with a dimension of 1×( +1), this vector contains global contextual sentiment semantic information and overall sentiment tendency.

[0107] Feature normalization eliminates differences in the numerical range of features across different modalities, avoiding imbalances in fusion weights caused by large voice feature values ​​and small text feature values. Temporal matching and cosine similarity checks ensure high temporal synchronization between the two modal features, reducing fusion errors caused by temporal misalignment. Linear interpolation fills in blank text feature segments, preventing the lack of corresponding text features for silent voice segments, ensuring the multimodal feature sequence covers the entire call duration. The verification process ensures feature trend consistency, reducing the impact of anomalies in a single modal feature on the overall sequence and enhancing feature reliability. Bidirectional attention calculation considers both voice-to-text and text-to-voice interaction, capturing bidirectional interactions between modalities better than unidirectional attention. The extracted deep semantic association features better align with actual emotional expression logic, improving the emotional discrimination capability of the fused features. Modal importance is determined based on acoustic confidence and semantic consistency indices, avoiding the problem of fixed weights being unsuitable for different scenarios. Voice weights are automatically increased to reduce text error impacts; conversely, text weights are increased to ensure weight allocation always matches modal reliability, enhancing the flexibility of sentiment analysis.

[0108] In a preferred embodiment of the present invention, step 5 above, which involves inputting the joint representation vector into the sentiment classifier to obtain the final sentiment category and intensity, may include:

[0109] Step 550: Input the joint representation vector into the pre-trained sentiment classifier. Calculate the probability distribution of sentiment categories using a multi-layer neural network. The sentiment categories include neutral, positive, and negative multi-level labels. Specifically, this includes the computational connection process of inputting the joint representation vector into the pre-trained sentiment classifier. First, obtain the joint representation vector, which contains multi-dimensional feature information of the text to be analyzed. Each element in the vector corresponds to a specific feature value. Use this joint representation vector as input data and transmit it to the pre-trained sentiment classifier. During the input process, the dimension of the joint representation vector is verified to ensure that its dimension is consistent with the dimension required by the input layer of the sentiment classifier. If the dimension does not match, it is adjusted through feature mapping calculation, such as using interpolation calculation. Based on the numerical patterns of the existing elements in the vector, the number of elements is supplemented or compressed to make the vector dimension meet the input requirements of the classifier. Then, the adjusted vector is input into the input layer of the classifier.

[0110] The process of calculating and generating the probability distribution of sentiment categories using a multi-layer neural network: The multi-layer neural network in the sentiment classifier includes an input layer, hidden layers, and an output layer. First, the input layer receives a joint representation vector after dimension adjustment and passes the value of each element in the vector to the first hidden layer. In the first hidden layer, each neuron performs a weighted calculation on the received value, that is, multiplies each input value by the corresponding weight parameter, and then sums all the weighted values ​​to obtain the initial input value of the neuron. Next, an activation function is applied to this initial input value for non-linear transformation calculation. Common activation functions such as the ReLU function will convert initial input values ​​less than 0 to 0, and initial input values ​​greater than 0 will remain unchanged. Through this calculation, the output value of each neuron in the first hidden layer is obtained.

[0111] Afterward, all the output values ​​of the first hidden layer are used as the input data for the second hidden layer. The above calculation process of weighting, summation, and activation function transformation is repeated, and so on, until the calculation of all hidden layers is completed. Finally, the output values ​​of the hidden layers are passed to the output layer. The output layer performs a softmax function calculation on the received values. The softmax function converts each output value into a probability value between 0 and 1, and the sum of all probability values ​​is 1. These probability values ​​correspond to the probability distribution of different sentiment categories such as neutral, positive, and negative. For example, a probability value of 0.6 means that the probability of the text belonging to the corresponding sentiment category is 60%.

[0112] Step 551, based on the probability distribution of sentiment categories, calculates the intensity value of sentiment through a regression branch. This specifically includes: obtaining basic data based on the probability distribution of sentiment categories. First, extract the probability distribution data of sentiment categories obtained in step 550, i.e., the probability values ​​corresponding to each category such as neutral, positive, and negative. Use these probability values ​​as the basic input data for the regression branch calculation, because these probability distributions reflect the degree of sentiment tendency in the text to a certain extent and can provide relevant basis for the calculation of the sentiment intensity value. The regression branch calculates the sentiment intensity value. The regression branch is also composed of a series of calculation units. First, perform feature transformation calculation on the input probability values, such as multiplying the probability values ​​of each category with preset weight coefficients. These weight coefficients are obtained by training on a large number of text samples labeled with sentiment intensity. They can adjust the influence of different sentiment categories on the intensity calculation according to their importance. Then, sum all the results of multiplication to obtain a preliminary feature comprehensive value.

[0113] Next, this initial feature synthesis value undergoes multiple linear transformations and nonlinear adjustment calculations. The linear transformation calculation multiplies the initial feature synthesis value by another weight matrix and adds a bias term to obtain a new value. The nonlinear adjustment calculation applies a suitable activation function, such as the tanh function, to map the value to the range of -1 to 1. This processes the linearly transformed value to better reflect the changing patterns of emotional intensity. After multiple rounds of such calculations, a linear calculation in the output layer finally maps the processed value to a preset emotional intensity range, such as 0 to 10, where 0 represents the lowest emotional intensity and 10 represents the highest emotional intensity, resulting in the final emotional intensity value. For example, a calculated value of 7.2 indicates that the emotional intensity of the text is 7.2.

[0114] Step 552 combines the probability distribution of sentiment categories with the sentiment intensity values ​​to obtain the final sentiment analysis result, including sentiment category labels and corresponding intensity values. Specifically, this includes: determining the sentiment category with the highest probability value in the sentiment category probability distribution; this category is the most likely sentiment category to which the text belongs; extracting its corresponding category label, such as "positive"; then, obtaining the sentiment intensity values ​​calculated in step 551; establishing a correspondence between the sentiment category label and the sentiment intensity value; ensuring that each determined sentiment category has a unique corresponding sentiment intensity value; integrating and calculating the established correspondence between the sentiment category labels and sentiment intensity values; for example, combining data according to the format of sentiment category labels and sentiment intensity values ​​to form structured result data; simultaneously, performing validity verification calculations on the integrated data; checking whether the sentiment category labels belong to the preset legal categories (neutral, positive, negative); and whether the sentiment intensity values ​​are within the preset reasonable range; if abnormal data is found, an error message is returned and the previous calculation steps are repeated; if all data is legal and valid, the integrated result is determined as the final sentiment analysis result, including sentiment category labels and corresponding intensity values.

[0115] By employing multi-layered neural networks for progressive weighting, summation, and activation function transformation calculations, the textual sentiment features inherent in the joint representation vector can be fully extracted. In particular, nonlinear transformation calculations can capture the complex nonlinear relationships within textual sentiment, resulting in a final sentiment category probability distribution that more closely reflects the actual emotional tendency of the text. This improves the accuracy of sentiment category judgment and reduces classification errors caused by the inability of simple linear calculations to capture complex emotional features. The regression branch, through multiple rounds of feature transformation, linear transformation, and nonlinear adjustment calculations on the sentiment category probability distribution data, can convert the degree of textual sentiment into a specific numerical form, achieving… Precise quantification of sentiment intensity, compared to simply judging sentiment category, allows sentiment intensity values ​​to more subtly reflect the strength of the text's emotions. For example, even with the same positive sentiment, intensity values ​​can distinguish between slightly positive and extremely positive, meeting the needs of scenarios requiring more refined sentiment analysis. Combining sentiment category labels with corresponding intensity values ​​to form structured final sentiment analysis results allows users to clearly understand the sentiment category of the text and the intensity of that sentiment at a glance. This avoids the information fragmentation that may occur when viewing sentiment category or sentiment intensity values ​​separately, providing a clear and intuitive presentation of results and reducing the difficulty for users to understand the sentiment analysis results.

[0116] like Figure 2 As shown, embodiments of the present invention also provide a call center dialogue sentiment analysis system that integrates voice and text, comprising:

[0117] The feature extraction module is used to acquire the voice signal and corresponding transcribed text of the call center, extract tone, speech rate, volume and pause emotion features from the voice signal, and perform deep language analysis on the transcribed text to extract semantic emotion features related to the context.

[0118] The association analysis module is used to perform cross-modal association analysis on speech sentiment features and semantic sentiment features. By constructing a correspondence analysis framework between feature sequences, it generates temporal correction coefficients for feature alignment.

[0119] The synchronization correction module is used to perform multimodal synchronous processing on speech emotion features and semantic emotion features using time-series correction coefficients, identify and correct time-series misalignment and semantic inconsistency caused by slips of the tongue, pauses and transcription errors, and generate a consistent multimodal feature sequence.

[0120] The fusion representation module is used to achieve deep fusion of speech sentiment features and semantic sentiment features based on multimodal feature sequences and cross-modal attention mechanism. It dynamically adjusts the influence of different modalities in sentiment analysis according to the confidence and semantic consistency of each modality, and generates a joint representation vector including global sentiment tendency.

[0121] The classification decision module is used to input the joint representation vector into the sentiment classifier to obtain the final sentiment category and intensity.

[0122] It should be noted that this system is a system corresponding to the above method. All implementation methods in the above method embodiments are applicable to this embodiment and can achieve the same technical effect.

[0123] Embodiments of the present invention also provide a computing device, including: a processor and a memory storing a computer program, wherein the computer program, when executed by the processor, performs the method described above. All implementations in the above method embodiments are applicable to this embodiment and can achieve the same technical effects.

[0124] Embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described above. All implementations in the above method embodiments are applicable to this embodiment and can achieve the same technical effects.

[0125] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for sentiment analysis of call center conversations fusing speech and text, characterized in that, The method comprises: obtaining voice signals and corresponding transcribed texts of a call center, extracting prosody, speech rate, intensity and pause emotional features from the voice signals, and simultaneously performing deep language analysis on the transcribed texts to extract context-related semantic emotional features; performing cross-modal correlation analysis on the voice emotional features and the semantic emotional features, generating time sequence correction coefficients for feature alignment by constructing a corresponding relationship analysis framework between the feature sequences; using the time sequence correction coefficients, performing multi-modal synchronous processing on the voice emotional features and the semantic emotional features, identifying and correcting the time sequence misplacement and semantic inconsistency problems caused by errors, pauses and transcription errors, and generating coordinated multi-modal feature sequences; based on the multi-modal feature sequences, using a cross-modal attention mechanism to realize deep fusion of voice emotional features and semantic emotional features, and dynamically adjusting the influence degree of different modalities in emotional analysis according to the confidence and semantic consistency of each modality, to generate a joint representation vector including a global emotional tendency; inputting the joint representation vector into an emotional classifier to obtain the final emotional category and intensity.

2. The fused voice and text call center conversation sentiment analysis method of claim 1, wherein, The cross-modal correlation analysis on the voice emotional features and the semantic emotional features is performed by constructing a corresponding relationship analysis framework between the feature sequences to generate time sequence correction coefficients for feature alignment, including: based on the voice emotional feature sequence and the semantic emotional feature sequence, combining the time stamp information of the voice frame and the text token to establish a fine-grained time sequence correspondence relationship between the two modal feature sequences; based on the fine-grained time sequence correspondence relationship, a cross-modal feature alignment framework is constructed to organize and map the two modal features in the shared time sequence dimension; based on the cross-modal feature alignment framework, the dynamic similarity measure and the correlation coefficient between the voice emotional features and the semantic emotional features in the same time interval are calculated; according to the dynamic similarity measure and the correlation coefficient, a set of time sequence correction coefficients for feature alignment is generated.

3. The fused voice and text call center conversation sentiment analysis method of claim 2, wherein, Based on the fine-grained time sequence correspondence relationship, a cross-modal feature alignment framework is constructed to organize and map the two modal features in the shared time sequence dimension, including: based on the time sequence correction coefficients, the dynamic alignment path between each frame in the voice feature sequence and each word unit in the text feature sequence is calculated to establish the time sequence mapping relationship from voice to text; according to the dynamic alignment path, the feature vector of each frame in the voice feature sequence is mapped to the time stamp position of the corresponding word unit in the text feature sequence to realize the projection and alignment of cross-modal features in the unified time sequence coordinates; in the unified time sequence coordinates, the mapped voice features and the corresponding text word unit features are aligned and spliced one by one to reorganize into a synchronous multi-modal fusion feature sequence.

4. The fused voice and text call center conversation sentiment analysis method of claim 3, wherein, Using the time sequence correction coefficients, the multi-modal synchronous processing is performed on the voice emotional features and the semantic emotional features to identify and correct the time sequence misplacement and semantic inconsistency problems caused by errors, pauses and transcription errors, and to generate coordinated multi-modal feature sequences, including: applying the time sequence correction coefficients to the voice emotional feature sequence and the semantic emotional feature sequence respectively to establish the mapping relationship between the two modal features; Based on the mapping relationship, the final alignment path is determined by calculating the cumulative distance between the two modal feature sequences, and the time scale adjustment is performed on the speech emotion feature sequence and the semantic emotion feature sequence to realize the time sequence alignment of the feature sequence; On the basis of time sequence alignment, the semantic inconsistency problem caused by errors in speech signal, such as mispronunciation, repetition and automatic speech recognition, is detected and corrected; The corrected speech emotion feature and semantic emotion feature are fused to generate a multi-modal feature sequence with consistent time sequence and coordination.

5. The fused voice and text call center conversation sentiment analysis method of claim 4, wherein, Based on the multi-modal feature sequence, a cross-modal attention mechanism is used to realize the deep fusion of speech emotion features and semantic emotion features, and according to the confidence of each modality and semantic consistency, the influence degree of different modalities in emotion analysis is dynamically adjusted to generate a joint representation vector including global emotional tendency, including: Cross-modal time sequence alignment processing is performed on the speech emotion feature sequence and the text emotion feature sequence to obtain the aligned multi-modal feature sequence; The aligned multi-modal feature sequence is input into the cross-modal attention mechanism, and the speech feature is taken as the query, the text feature is taken as the key value, and the text feature is taken as the query, the speech feature is taken as the key value, respectively, to perform bidirectional attention calculation, realize the interaction modeling between modalities, extract deep semantic correlation features, and obtain cross-modal fusion features; Based on the cross-modal fusion features, the acoustic confidence index of the speech feature and the semantic consistency index of the text feature are calculated respectively, and the importance of the speech modality and the text modality in emotion recognition is dynamically determined according to the two types of indexes; Based on the importance of the modality, the cross-modal fusion features are adaptively fused to generate a joint emotion representation vector including global context emotional semantic information.

6. The fused voice and text call center conversation sentiment analysis method of claim 5, wherein, The joint representation vector is input into the emotion classifier to obtain the final emotion category and intensity, including: The joint representation vector is input into the pre-trained emotion classifier, and the probability distribution of the emotion category is generated through multi-layer neural network calculation; Based on the probability distribution of the emotion category, the intensity value of the emotion is calculated through the regression branch; The probability distribution of the emotion category and the intensity value of the emotion are combined to obtain the final emotion analysis result including the emotion category label and the corresponding intensity value.

7. The fused voice and text call center conversation sentiment analysis method of claim 6, wherein, The emotion category includes neutral, positive and negative multi-level labels.

8. A call center conversational sentiment analysis system that fuses speech and text, the system implementing the method of any one of claims 1 to 7, characterized in that, Including: The feature extraction module is used to obtain the speech signal and the corresponding transcribed text of the call center, extract the intonation, speech rate, sound intensity and pause emotion features from the speech signal, and perform deep language analysis on the transcribed text to extract context-related semantic emotion features; The correlation analysis module is used for cross-modal correlation analysis of speech emotion features and semantic emotion features, and a corresponding relationship analysis framework between feature sequences is constructed to generate time sequence correction coefficients for feature alignment; The synchronous correction module is used to perform multi-modal synchronous processing on speech emotion features and semantic emotion features by using the time sequence correction coefficients, identify and correct the time sequence misalignment and semantic inconsistency problems caused by mispronunciation, pause and transcription errors, and generate a multi-modal feature sequence with consistent coordination. The fusion representation module is configured to realize deep fusion of the speech emotion feature and the semantic emotion feature based on the multi-modal feature sequence by using a cross-modal attention mechanism, and dynamically adjust an influence degree of different modalities in emotion analysis according to a confidence of each modality and semantic consistency, and generate a joint representation vector including a global emotional tendency. The classification decision module is configured to input the joint representation vector into an emotion classifier to obtain a final emotion category and intensity.

9. A computing device, comprising: The method comprises the following steps: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a program, and the program is executed by the processor to implement the method according to any one of claims 1 to 7.

Images