User emotion recognition method and device, computer device, and storage medium

Abstract

This invention relates to the field of artificial intelligence technology and discloses a user emotion recognition method, apparatus, computer device, and storage medium, comprising: acquiring and parsing associated call recording data to extract speech data and corresponding text data; performing speech frame processing on the speech data to obtain corresponding frequency domain features, and extracting features from the speech frame feature sequence composed of the frequency domain features corresponding to each speech frame using a preset speech emotion feature encoder to generate speech features; performing text segmentation on the text data to obtain corresponding text semantic features, and extracting features from the text unit feature sequence composed of the text unit semantic features corresponding to each text unit using a preset text emotion feature encoder to generate text features; generating fused features based on speech features and text features, and identifying the emotion classification result. This invention can be applied to user emotion recognition scenarios in financial technology and can effectively improve the accuracy of user emotion recognition.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and specifically to a user emotion recognition method, apparatus, computer device, and computer-readable storage medium. Background Technology

[0002] Currently, in the property insurance financial business scenarios of fintech, insurance institutions' customer service call centers handle a large amount of business communication work, including claim acceptance, claims consultation, policy consultation, and complaint handling. The recorded call data between target users and insurance customer service is a crucial data carrier reflecting the target user's business needs, true intentions, and emotional state. By conducting user sentiment recognition on these call recordings, insurance institutions can, on the one hand, perceive changes in the target user's emotions, identify potential high-risk complaints in advance, optimize claims and policy service processes, and improve customer service quality and risk management efficiency in property insurance business; on the other hand, it can also combine emotional information to help uncover the target user's true intentions, distinguish between the user's apparent statements and their true inner needs, and provide a basis for claim analysis, dispute prediction, and business solution matching.

[0003] Currently, existing technologies for user emotion recognition in call recordings mainly fall into two categories: manual recognition and automated single-modal recognition. Manual sampling recognition is costly, has limited coverage, and suffers from significant subjective errors, failing to meet the analysis needs of full-volume call data. Mainstream automated emotion recognition largely employs single-modal recognition, relying solely on a single voice or text modality to determine emotion. Specifically, while recognition based solely on voice acoustic signals can capture acoustic emotion information such as tone and speed, it cannot identify the specific business semantics of the call content, making it difficult to accurately assess users' true needs and emotional motivations in insurance business scenarios. Semantic recognition based solely on speech-to-text can parse the text semantics and identify business needs, but it completely loses key acoustic emotion information such as tone and emotional fluctuations, easily leading to semantic matching but distorted emotion judgment, failing to accurately reconstruct the user's call state. Single-modal information has inherent limitations, lacking information dimensions and sufficient representational capabilities, ultimately resulting in low accuracy in user emotion recognition and an inability to accurately meet the needs of user emotion recognition and true intent assessment in property insurance business scenarios.

[0004] Therefore, how to provide a user emotion recognition method, device, computer equipment, and computer-readable storage medium that can effectively improve the accuracy of user emotion recognition is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the purpose of this invention is to provide a user emotion recognition method, apparatus, computer device and computer-readable storage medium, aiming to solve the problem of how to effectively improve the accuracy of user emotion recognition.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a user emotion recognition method, comprising: Obtain call recording data associated with the target user, parse the call recording data, and extract voice data and text data corresponding to the voice data from the parsing results; The speech data is processed by speech framing. Based on the speech frames obtained after speech framing, the corresponding frequency domain features are obtained. Through a preset speech emotion feature encoder, the speech frame feature sequence composed of the frequency domain features corresponding to each speech frame is extracted to generate the speech features of the speech data. The text data is processed by text segmentation. Based on the text units obtained after text segmentation, the corresponding text semantic features are obtained. Through a preset text sentiment encoder, the text unit feature sequence composed of the text semantic features corresponding to each text unit is extracted to generate the text features of the text data. Based on the voice features and the text features, a fusion feature of the call recording data is generated, and based on the fusion feature, the emotion classification result of the target user is identified.

[0007] Secondly, the present invention provides a user emotion recognition device, comprising: The acquisition module is used to acquire call recording data associated with the target user, parse the call recording data, and extract voice data and text data corresponding to the voice data from the parsing results. The speech framing module is used to perform speech framing processing on the speech data, obtain the corresponding frequency domain features based on the speech frames obtained after speech framing, and extract features from the speech frame feature sequence composed of the frequency domain features corresponding to each speech frame through a preset speech emotion feature encoder to generate the speech features of the speech data. The text segmentation module is used to perform text segmentation processing on the text data, obtain the corresponding text semantic features based on the text units obtained after text segmentation, and extract features from the text unit feature sequence composed of the text semantic features corresponding to each text unit through a preset text sentiment feature encoder to generate the text features of the text data. The recognition module is used to generate fusion features of the call recording data based on the voice features and the text features, and to identify the emotion classification result of the target user based on the fusion features.

[0008] Thirdly, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the user emotion recognition method as described above.

[0009] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the user emotion recognition method described above.

[0010] Compared to existing technologies, this invention provides a user emotion recognition method, apparatus, computer device, and computer-readable storage medium, comprising: acquiring call recording data associated with a target user; parsing the call recording data; extracting voice data and corresponding text data from the parsing results; performing voice frame processing on the voice data; obtaining corresponding frequency domain features based on the voice frames obtained after voice frame processing; extracting features from the voice frame feature sequence composed of the frequency domain features corresponding to each voice frame using a preset voice emotion feature encoder to generate voice features of the voice data; performing text segmentation on the text data; obtaining corresponding text semantic features based on the text units obtained after text segmentation; extracting features from the text unit feature sequence composed of the text semantic features corresponding to each text unit using a preset text emotion feature encoder to generate text features of the text data; generating fusion features of the call recording data based on the voice features and the text features; and identifying the emotion classification result of the target user based on the fusion features. This invention generates fusion features from call recording data by integrating speech and text features, and performs emotion recognition by combining two complementary types of information, acoustic and semantic dimensions, which can effectively improve the accuracy of emotion classification results for target users. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1This is a schematic diagram illustrating the application environment of a user emotion recognition method provided in an embodiment of the present invention.

[0013] Figure 2 This is a flowchart illustrating a user emotion recognition method according to an embodiment of the present invention.

[0014] Figure 3 for Figure 2 A schematic diagram of the specific process of step S100.

[0015] Figure 4 for Figure 2 A schematic diagram of the specific process for step S200.

[0016] Figure 5 for Figure 2 A schematic diagram of the specific process of step S300.

[0017] Figure 6 for Figure 2 A schematic diagram of the specific process of step S400.

[0018] Figure 7 This is a schematic diagram of the program modules of a user emotion recognition device provided in an embodiment of the present invention.

[0019] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention.

[0020] Figure 9 This is another structural schematic diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0023] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0024] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0025] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0026] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0027] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0028] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0029] An embodiment of the present invention provides a user emotion recognition method, which can be applied to, for example... Figure 1In the application environment shown, the client and server communicate via a network. The client includes, but is not limited to, handheld computers, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud computing devices, and personal digital assistants (PDAs). The server can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0030] Please see Figure 2 An embodiment of the present invention provides a user emotion recognition method, wherein the method includes the following steps: S100: Obtain call recording data associated with the target user, parse the call recording data, and extract voice data and text data corresponding to the voice data from the parsing result; S200. Perform speech framing processing on the speech data, obtain the corresponding frequency domain features based on the speech frames obtained after speech framing, and extract features from the speech frame feature sequence composed of the frequency domain features corresponding to each speech frame through a preset speech emotion feature encoder to generate the speech features of the speech data. S300. Perform text segmentation processing on the text data, obtain the corresponding text semantic features based on the text units obtained after text segmentation, and extract features from the text unit feature sequence composed of the text semantic features corresponding to each text unit through a preset text sentiment encoder to generate the text features of the text data. S400. Based on the voice features and the text features, generate fusion features of the call recording data, and identify the emotion classification result of the target user according to the fusion features.

[0031] In specific implementation, the user emotion recognition method of this embodiment obtains both voice data and text data from the same source based on the call recording data. On one hand, the voice data is processed by voice framing to obtain voice frames. Based on the voice frames, the corresponding frequency domain features are extracted to construct a voice frame feature sequence. Then, a preset voice emotion feature encoder is used to complete the acoustic dimension feature extraction to obtain voice features, which can fully explore the emotion-related acoustic information in the call audio, such as tone and speech rate, which cannot be reflected in the text. On the other hand, the text data corresponding to the voice data is processed by text segmentation to obtain text units. Based on the text units, the corresponding text semantic features are obtained to construct a text unit feature sequence. The method then mines the emotional information at the semantic level of the text through a pre-set text sentiment encoder to obtain text features, making full use of the business semantic content carried by the transcribed text. This method does not rely solely on single-modal information of speech or text, but rather integrates speech features and text features to generate fusion features of call recording data. It combines two complementary information dimensions, acoustic dimension and semantic dimension, to carry out sentiment recognition, avoiding the shortcomings of single-modal solutions where speech is easily affected by environmental noise and text is easily affected by transcription errors. By comprehensively utilizing the effective information of both modalities, it reduces the recognition bias caused by the defects of a single modality, thereby effectively improving the recognition accuracy of sentiment classification results for target users.

[0032] Understandably, the user emotion recognition method provided in this embodiment of the invention can be applied to user emotion recognition scenarios related to the fintech field. The following is a specific example: In a property insurance customer service call center scenario, when an insurance institution receives a claim report call from a target user, it obtains the corresponding call recording data. After parsing the call recording data, it obtains both voice data and corresponding transcribed text data. On one hand, the voice data is processed by frame segmentation, extracting frequency domain features to construct a voice frame feature sequence. This sequence is then used to extract voice features via a voice emotion encoder, capturing acoustic information such as the target user's tone and emotional fluctuations. On the other hand, the transcribed text data is processed by text segmentation to obtain text units. Semantic features are extracted to construct a text unit feature sequence, which is then used to extract text features via a text emotion encoder, obtaining business semantic information such as the claim request and event description. The voice and text features are then fused to obtain a fused feature, which is used to identify the target user's emotional classification. Based on the identified emotional classification and the business content carried by the call text, the insurance institution can identify the target user's emotional state, such as anxiety or dissatisfaction, providing early warnings of complaint risks. It can also help determine the target user's true business intentions, providing a reference for subsequent claims handling, dispute resolution, and service resource allocation.

[0033] Furthermore, in one embodiment, the user emotion recognition method, wherein step S100, acquiring call recording data associated with the target user, parsing the call recording data, and extracting voice data and corresponding text data from the parsing result, specifically includes: S110. Obtain call recording data associated with the target user, perform validity verification on the call recording data, remove invalid recording segments with signal amplitude lower than a preset signal threshold, and obtain verified call recording data. S120. Perform audio parsing and separation processing on the verified call recording data to obtain the original audio stream, and perform noise suppression processing on the original audio stream to obtain noise-reduced voice data. S130. The denoised speech data is transcribed to generate transcribed text, and a timestamp mapping relationship is established between the transcribed text and the denoised speech data to obtain text data that is time-aligned with the denoised speech data.

[0034] In specific implementation, after obtaining the call recording data associated with the target user, this embodiment first performs a recording validity check, removing invalid recording segments with signal amplitudes below a preset signal threshold and filtering out invalid segments without valid human voices to reduce the interference of invalid data on subsequent processing. Then, the verified call recording data undergoes audio parsing and separation as well as noise suppression processing to reduce the interference of environmental noise on the speech signal and improve the signal quality of the speech data itself. At the same time, the denoised speech data is transcribed to obtain transcribed text, and a timestamp mapping relationship is established between the transcribed text and the denoised speech data to achieve time alignment between text data and speech data. This ensures the accuracy of the temporal correspondence between speech data and text data, providing reliable and temporally matched input data for subsequent bimodal feature extraction and fusion recognition, reducing recognition errors caused by abnormal recording signals, environmental noise, and temporal misalignment, and further ensuring the reliability of subsequent user emotion recognition results.

[0035] The specific implementation process of this embodiment is roughly as follows: 1. Verification of the validity of call recording data: The system acquires call recording data associated with the target user and segments the complete call recording data into multiple continuous recording segments according to a fixed time slice granularity. It calculates the audio signal amplitude of each recording segment and compares the signal amplitude of the recording segment with a pre-configured preset signal threshold. Recording segments with signal amplitudes lower than the preset signal threshold are identified as invalid recording segments lacking valid human voices and are directly removed. All remaining valid recording segments after removing invalid segments are spliced ​​together according to the original call's chronological order to obtain the verified call recording data. This filters out invalid segments generated during the call, such as prolonged silence, signal interruption, and no sound output, preventing invalid segments from consuming processing resources in subsequent algorithms.

[0036] 2. Audio parsing and separation processing: The system reads the file encapsulation format of the verified call recording data, and parses and removes the file header, metadata, channel markers, and other encapsulation layer information from the call recording data. Through audio parsing and separation processing, it extracts the original audio stream that has not undergone audio modification from the encapsulated file. If the call recording data is a stereo recording, it separates the original mono audio stream corresponding to the target user, filters out the audio data of the customer service side channel, and retains only the audio data stream corresponding to the target user's speech, providing the original audio input for subsequent noise suppression processing.

[0037] 3. Noise suppression processing of the original audio stream: The original audio stream is segmented into frames, and the noise estimate of each frame is calculated to identify background noise from the call environment and channel noise introduced by line transmission. Noise suppression filter coefficients are generated based on the noise estimates, and these coefficients are used to attenuate and suppress the noise components in the original audio stream while protecting the spectral components corresponding to human voice from excessive attenuation. After noise suppression is completed, the noise-reduced speech data is output, reducing the interference of environmental noise on the human voice signal and improving the signal-to-noise ratio of the human voice signal.

[0038] 4. Speech transcription and timestamp mapping construction: The denoised speech data is fed into a speech-to-text model, which recognizes the speech content frame by frame and outputs complete transcribed text. It also outputs the audio start and end timestamps for each transcribed text segment. Using these audio timestamps as a link, a timestamp mapping relationship is established between the transcribed text segments and the audio segments of the denoised speech data. Based on this mapping relationship, text data aligned with the time of the denoised speech data is obtained, ensuring that any speech signal can be quickly located to its corresponding transcribed text content. This provides a time-aligned data foundation for subsequent simultaneous feature extraction in both speech and text modalities.

[0039] Further, in one embodiment, the user emotion recognition method, wherein step S200, involves performing speech frame segmentation on the speech data, obtaining corresponding frequency domain features based on the speech frames obtained after speech frame segmentation, and extracting features from the speech frame feature sequence composed of the frequency domain features corresponding to each speech frame using a preset speech emotion feature encoder to generate speech features of the speech data, specifically includes: S210. The speech data is divided into frames according to the preset frame length to obtain multiple speech frames. The validity of each speech frame is screened according to the preset frame quality threshold. Invalid speech frames with noise ratio exceeding the preset frame quality threshold are removed, and valid speech frames are retained. S220. Windowing and short-time Fourier transform are performed on each of the effective speech frames in sequence to obtain the corresponding frame spectrum. The frame spectrum is mapped using a Mel filter bank to obtain the Mel spectrum. The Mel spectra corresponding to each of the effective speech frames are spliced ​​together to obtain the Mel spectrum sequence. S230. Using a preset speech emotion feature encoder, feature extraction is performed on the Mel spectrum sequence to generate speech features of the speech data.

[0040] Furthermore, the user emotion recognition method, wherein the step of sequentially performing windowing processing and short-time Fourier transform on each of the effective speech frames to obtain the corresponding frame spectrum, mapping the frame spectrum using a Mel filter bank to obtain a Mel spectrum, and concatenating the Mel spectra corresponding to each of the effective speech frames to obtain a Mel spectrum sequence, specifically includes: Windowing is applied to each of the effective speech frames, and a short-time Fourier transform is performed on each of the windowed effective speech frames to obtain the corresponding frame spectrum. Then, the amplitude of each frame spectrum is normalized according to a preset amplitude range to obtain the normalized frame spectrum. The normalized frame spectra are mapped using a Mel filter bank to obtain the Mel spectrum corresponding to each effective speech frame. The corresponding Mel spectrum is then dynamically weighted at the frame level based on the frame quality score of each effective speech frame to obtain the weighted Mel spectrum. The weighted Mel spectra corresponding to each of the effective speech frames are concatenated to obtain a Mel spectrum sequence.

[0041] In specific implementation, this embodiment first performs frame segmentation of the speech data according to a preset frame length, and then performs validity screening of the speech frames based on a preset frame quality threshold, removing invalid speech frames with a noise ratio exceeding the preset frame quality threshold, and retaining valid speech frames, which can filter out interference caused by high-noise speech frames; then, windowing processing and short-time Fourier transform are performed on each valid speech frame to obtain the frame spectrum, and then amplitude normalization processing is performed to obtain the normalized frame spectrum, unifying the spectral amplitude distribution between different speech frames and reducing the impact of recording volume fluctuations; then, Mel filter bank is used to complete the spectrum mapping to obtain the Mel spectrum, and based on each valid speech frame... The frame quality score of the audio frame is used to perform frame-level dynamic weighting of the corresponding Mel spectrum, which reduces the contribution of poor-quality effective speech frames to the spectral representation and suppresses the negative impact of residual noise on spectral features. Finally, the weighted Mel spectra corresponding to all effective speech frames are concatenated to form a Mel spectrum sequence, which is then fed into a preset speech emotion feature encoder for feature extraction to generate speech features. This improves the data quality of the spectral sequence input to the encoder, reduces the interference of noise and volume fluctuations on acoustic feature extraction, and enables the output speech features to more realistically reflect the acoustic emotion information of the target user, thereby improving the accuracy of subsequent user emotion recognition results.

[0042] The specific implementation process of this embodiment is roughly as follows: 1. Speech frame segmentation and speech frame validity screening: The system reads the pre-denoised speech data and segments the continuous time-domain speech signal according to a pre-configured frame length. Simultaneously, it configures a corresponding pre-configured frame shift to achieve signal overlap between adjacent speech frames, thus ensuring the continuity of the speech signal in the time domain. After segmentation, multiple speech frames are obtained. For each speech frame, the ratio of intra-frame noise energy to total signal energy is calculated to obtain the noise percentage of that speech frame. The calculated noise percentage is compared with a pre-configured frame quality threshold. Speech frames with a noise percentage higher than the pre-configured frame quality threshold are determined as invalid speech frames and discarded. Speech frames with a noise percentage not exceeding the pre-configured frame quality threshold are marked as valid speech frames and retained. A frame quality score is recorded for each valid speech frame, reflecting the signal purity of a single speech frame. High-noise invalid speech frames are removed to prevent strong noise frames from participating in subsequent frequency domain feature calculations and reduce feature contamination caused by noise.

[0043] 2. Effective speech frame windowing processing, short-time Fourier transform, and spectral amplitude normalization: For each valid speech frame selected and retained, a window function is applied to perform windowing processing. The window function attenuates the two ends of the speech frame signal, mitigating the spectral leakage caused by directly truncating the time-domain signal. For each valid speech frame that has undergone windowing processing, a short-time Fourier transform is performed to convert the time-domain speech signal to the frequency domain, outputting the original frame spectrum for each frame. The amplitude of the original frame spectrum can fluctuate significantly due to the volume of the call device and the gain of the recording channel. Therefore, according to a preset amplitude range, the amplitude of all frequency points of each original frame spectrum is scaled and mapped to complete the amplitude normalization processing, compressing the amplitude differences under different recording conditions, and obtaining the normalized frame spectrum to ensure that the input of subsequent spectrum mapping is within a uniform numerical range.

[0044] 3. Mel filter bank mapping and Mel spectrum frame-level dynamic weighting: The normalized frame spectrum is input into a Mel filter bank. A set of triangular bandpass filters is used to map the spectrum of the linear frequency scale to the Mel frequency scale, simulating the auditory perception characteristics of the human ear for different frequencies of sound, and the original Mel spectrum corresponding to each effective speech frame is calculated. The frame quality score of the corresponding effective speech frame calculated in the previous step is retrieved, and the frame quality score is used as a weighting coefficient to perform frame-level dynamic weighting operation on all dimensions of the original Mel spectrum. For effective speech frames with low frame quality scores and a small amount of residual noise, the overall weight of the Mel spectrum of that frame is reduced. For effective speech frames with high frame quality scores and good voice purity, a higher weight contribution is retained, thereby suppressing the interference of residual noise in the effective speech frame on the features, and the weighted Mel spectrum is obtained.

[0045] 4. Mel spectrum sequence splicing and speech emotion feature encoder feature extraction: The weighted Mel spectra after processing are concatenated and combined strictly according to the original temporal order of the corresponding valid speech frames to construct a complete Mel spectrum sequence. This sequence fully preserves the acoustic information of the speech signal as it changes over time. The constructed Mel spectrum sequence is then input into a preset speech emotion feature encoder. The speech emotion feature encoder performs deep extraction of the temporal correlation and acoustic emotion-related hidden features within the sequence, compresses the sequence dimension, and outputs speech features of fixed-dimensional speech data. The output speech features carry acoustic emotion information such as tone and emotional fluctuations, providing a reliable modal input for subsequent multimodal fusion with text features.

[0046] Further, in one embodiment, the user sentiment recognition method, wherein step S300, involves performing text segmentation on the text data, obtaining corresponding text semantic features based on the text units obtained after text segmentation, and extracting features from the text unit feature sequence composed of the text semantic features corresponding to each text unit through a preset text sentiment feature encoder to generate text features of the text data, specifically includes: S310. Perform text segmentation processing on the text data to obtain multiple text units, and perform confidence screening on each text unit according to a preset transcription confidence threshold to obtain a reliable text unit. S320. Each of the trusted text units is encoded by word, sentence and position to obtain the corresponding text semantic features. A corresponding text unit semantic mask is generated based on the transcription confidence of each of the trusted text units. A text unit feature sequence is constructed based on the text semantic features and the text unit semantic mask. S330. Using a preset text sentiment encoder, feature extraction is performed on the feature sequence of the text unit to generate the text features of the text data.

[0047] Furthermore, in the aforementioned user emotion recognition method, the step of performing text segmentation on the text data to obtain multiple text units, and then filtering the credibility of each text unit according to a preset transcription confidence threshold to obtain credible text units, specifically includes: The text data is segmented using a preset word segmenter, and special symbols and punctuation marks in the text data are removed to obtain multiple sub-word units after splitting. The multiple sub-word units are arranged in the original chronological order of the text data to form a sub-word unit sequence. A classifier is added at the beginning of the sub-word unit sequence and a separator is added at the end to form a text unit sequence. Each element in the text unit sequence is taken as a text unit to obtain multiple text units. Obtain the transcription confidence level corresponding to each text unit, and perform confidence screening on each text unit according to the preset transcription confidence level threshold. Remove low-confidence text units with transcription confidence levels lower than the preset transcription confidence level threshold, and retain the reliable text units.

[0048] In specific implementation, this embodiment first uses a preset word segmenter to segment the text data, removing special symbols and punctuation marks, and then adds classifiers and delimiters to obtain multiple text units. Based on a preset transcription confidence threshold, a confidence screening process is performed, eliminating low-confidence text units with low transcription confidence and retaining reliable text units. This filters out unreliable text content generated by speech-to-text errors, preventing erroneous transcription content from interfering with semantic analysis. Next, each reliable text unit is encoded with word, sentence, and position to obtain corresponding text semantic features. Simultaneously, text units are generated by combining the transcription confidence of the reliable text units. Semantic masking constrains the semantic contribution of text units with different confidence levels, reducing the influence of poorly transcribed text units on features, thereby constructing a text unit feature sequence. This text unit feature sequence is then input into a preset text sentiment feature encoder to complete feature extraction, generating text features of the text data. This improves the data quality of the text unit feature sequence input to the encoder, reduces the negative impact of speech transcription errors, and ensures that the obtained text features can accurately reflect the business semantics and sentiment information of the call content, guaranteeing the reliability of subsequent multimodal fusion and sentiment classification recognition results.

[0049] The specific implementation process of this embodiment is roughly as follows: 1. Text segmentation and purification, and sub-word unit splitting: For complete text data aligned with the speech time sequence, a preset word segmenter is invoked to perform fine-grained sub-word segmentation. The entire call text is segmented word by word and character by character according to the semantic granularity of insurance business text. At the same time, all segmentation results are traversed, and invalid special symbols, redundant spaces, various punctuation marks, and redundant characters without business semantics are uniformly filtered and removed from the text. Only valid words with actual semantics are retained. Finally, multiple clean and non-redundant sub-word units are obtained, ensuring that subsequent semantic learning is not interfered with by invalid characters.

[0050] 2. Sub-word sequence recombination and standard text unit construction: All purified sub-word units are arranged in strict chronological order according to the original text of the user's call, forming a continuous and complete sub-word unit sequence. A preset classifier is uniformly added at the beginning of the overall sub-word unit sequence, and a preset separator is uniformly added at the end of the sequence to complete the encapsulation of the beginning and end marks of the sequence, thus constructing a text unit sequence with a regular structure and unified beginning and end marks. Each independent element in this text unit sequence is defined as an independent text unit, completing the structured generation of all text units, thereby obtaining all the multiple text units to be processed.

[0051] 3. Confidential text unit filtering based on transcription confidence: The system retrieves the transcription confidence scores of each text unit output synchronously during the speech-to-text stage. These scores quantify the probability that the model accurately recognizes the word. The transcription confidence score of each text unit is compared and screened against a preset transcription confidence threshold. Low-confidence text units with transcription confidence scores below the threshold, or those exhibiting transcription errors, omissions, or ambiguous recognition are removed entirely. This process thoroughly eliminates erroneous text content caused by insurance terminology, accents, and noise interference. Only reliable text units with acceptable transcription confidence scores, accurate recognition, and semantic reliability are retained, achieving precise noise reduction and purification of the text data.

[0052] 4. Multi-dimensional encoding and confidence-adaptive semantic mask generation: For each selected trusted text unit, three types of encoding processes are performed in parallel: word encoding extracts the semantic features of individual characters, sentence encoding captures the overall semantics of the context, and position encoding records the temporal position and word order information of the text unit in the entire text. The three types of encoding are fused to generate text semantic features that accurately represent the content of the unit. At the same time, a dedicated semantic mask is dynamically generated based on the transcription confidence of each trusted text unit. Strong constraint masks are set for trusted text units with relatively low confidence to weaken their semantic weight, while the complete semantic weight is retained for trusted text units with high confidence, so as to achieve flexible suppression of slight transcription errors.

[0053] 5. Feature sequence construction and text sentiment feature encoding extraction: By combining the text semantic features corresponding to all trusted text units with their corresponding text unit semantic masks and arranging them in a regular manner according to the original temporal order of the text, a text unit feature sequence with temporal integrity, semantic reliability, and built-in error constraint capability is constructed. The constructed text unit feature sequence is then fed into a preset text sentiment feature encoder, which deeply mines the text context semantic features, user demand features, and implicit sentiment features, automatically completes feature aggregation and dimensional optimization, and finally outputs high-precision text features corresponding to the overall text data, providing pure and reliable text modality feature support for subsequent dual-modal feature fusion and sentiment recognition.

[0054] Further, in one embodiment, the user emotion recognition method, wherein step S400, generating fusion features of the call recording data based on the voice features and the text features, and identifying the emotion classification result of the target user based on the fusion features, specifically includes: S410. Perform cross-modal feature alignment calibration on the speech features and the text features to obtain dimension-aligned target speech features and target text features. S420. Aggregate and calculate the frame quality scores corresponding to each valid speech frame in the speech data to obtain the speech-side frame quality score. Aggregate and calculate the transcription confidence scores corresponding to each reliable text unit in the text data to obtain the text-side transcription confidence scores. S430. Based on the target speech features, the target text features, the speech-side frame quality score, and the text-side transcription confidence, perform multimodal weighted fusion to generate the fusion features of the call recording data; S440. Perform sentiment inference and recognition based on the fused features to generate the sentiment classification result for the target user.

[0055] In specific implementation, this embodiment first performs cross-modal feature alignment calibration on speech and text features to eliminate the inconsistency between the dimensions and representation spaces of the two types of modal features, ensuring that speech and text features can be accurately matched and fused. Then, it aggregates and calculates the frame quality scores corresponding to each effective speech frame and the transcription confidence scores corresponding to each reliable text unit obtained from the pre-process, obtaining the speech-side frame quality score and text-side transcription confidence score that can characterize the overall signal quality and transcription confidence of the entire call data, achieving precise quantification from fine-grained unit quality to global modal quality. Finally, based on the global quality scores of the two modalities, it analyzes the aligned target speech features and text features... Adaptive multimodal weighted fusion is performed on the standard text features. The fusion weights are dynamically allocated according to the data quality of the voice and text, automatically weakening the feature contribution of low-quality modalities and strengthening the effective feature information of high-confidence modalities. This effectively suppresses feature interference caused by call noise and transcription errors, and avoids the shortcomings of fixed-weight fusion methods that cannot adapt to the differences in call data quality. Finally, sentiment inference recognition is completed based on highly robust fusion features. By fully combining acoustic sentiment information and text semantic information, the accuracy and stability of user sentiment classification results in property insurance call scenarios are significantly improved, while the accuracy of recognizing users' true emotions and business intentions is also improved.

[0056] The specific implementation process of this embodiment is roughly as follows: 1. Cross-modal feature alignment calibration: The system reads speech features output by the speech emotion encoder and text features output by the text emotion encoder. Since speech features and text features are extracted by two independent encoders, their original feature dimensions are different, and their feature representation spaces are independent, making direct fusion computation impossible. A pre-defined cross-modal mapping network is used to perform projection transformations on speech features and text features respectively, mapping them to the same common feature representation space and unifying the vector dimensions of the two types of features. After the mapping transformation, the system outputs dimension-aligned target speech features and target text features, eliminating the problems of dimension mismatch and inconsistent representation spaces between different modalities, and ensuring that vector operations can be executed normally in the subsequent multimodal weighted fusion process.

[0057] 2. Modal global quality score aggregation calculation: The process involves retrieving frame quality scores from all valid speech frames saved in the previous speech processing stage. These scores represent the signal purity of each valid speech frame. A weighted average is calculated on the frame quality scores of all valid speech frames, with weights set according to the duration of each frame, to obtain a speech-side frame quality score reflecting the overall signal-to-noise ratio of the entire speech segment. Simultaneously, the process retrieves transcription confidence scores from all reliable text units retained in the text processing stage. These confidence scores represent the reliability of transcription and recognition of each word unit. The average of the transcription confidence scores of all reliable text units is calculated to obtain a text-side transcription confidence score representing the overall accuracy of the transcribed text. This completes the conversion from fine-grained quality indicators for single frames and single text units to global modal quality indicators for the entire speech segment, resulting in a global quality quantification value that can guide the allocation of fusion weights.

[0058] 3. Adaptive multimodal weighted fusion based on modal quality: The speech frame quality score and text transcription confidence score are input into the weight calculation logic. The two quality scores are normalized to obtain the speech fusion weight corresponding to the target speech feature and the text fusion weight corresponding to the target text feature. When the speech frame quality score is low, indicating significant residual speech noise, the speech fusion weight is automatically reduced, weakening the contribution of the target speech feature in the fusion process. Similarly, when the text transcription confidence score is low, indicating a high overall error probability in the transcribed text, the text fusion weight is automatically reduced, weakening the contribution of the target text feature in the fusion process. The dimension-aligned target speech feature and target text feature are multiplied by their respective dynamic fusion weights, and then feature vector merging is performed to generate the fusion feature corresponding to the call recording data. This achieves adaptive adjustment of the contribution ratio of each modality feature based on the actual data quality of the two modalities, avoiding the shortcomings of fixed-weight fusion which cannot adapt to different call recording qualities.

[0059] 4. Feature-based sentiment reasoning and recognition: The calculated fusion features are input into a pre-trained sentiment reasoning and recognition model. The sentiment reasoning and recognition model mines the acoustic emotion information and business semantic information fused together in the fusion features and completes the feature discrimination operation. The model internally goes through fully connected layer mapping and classification layer calculation, and outputs the sentiment classification result corresponding to the target user according to the predefined sentiment category of property insurance business. The sentiment classification result comprehensively utilizes the effective information of high-confidence modalities and suppresses the interference information introduced by low-quality modalities. It can not only reflect the emotional state of the target user, but also help support the subsequent judgment of the target user's true business intention.

[0060] As can be seen from the above method embodiments, the user emotion recognition method provided by the present invention includes: acquiring call recording data associated with a target user; parsing the call recording data; extracting voice data and text data corresponding to the voice data from the parsing result; performing voice frame processing on the voice data; obtaining corresponding frequency domain features based on the voice frames obtained after voice frame processing; extracting features from the voice frame feature sequence composed of the frequency domain features corresponding to each voice frame using a preset voice emotion feature encoder to generate voice features of the voice data; performing text segmentation processing on the text data; obtaining corresponding text semantic features based on the text units obtained after text segmentation; extracting features from the text unit feature sequence composed of the text semantic features corresponding to each text unit using a preset text emotion feature encoder to generate text features of the text data; generating fusion features of the call recording data based on the voice features and the text features; and identifying the emotion classification result of the target user based on the fusion features. The method of this invention generates fusion features from call recording data by fusing speech features and text features, and performs emotion recognition by integrating two complementary types of information, acoustic dimension and semantic dimension, which can effectively improve the recognition accuracy of emotion classification results for target users.

[0061] It should be understood that although this application provides the method operation steps as described in the embodiments or flowcharts, conventional or non-inventive labor may include more or fewer operation steps, and these operation steps are not necessarily executed sequentially according to the order of the embodiments or flowcharts. The order of steps listed in the embodiments or flowcharts is merely one way of executing many steps and does not represent the only execution order. It should be noted that there is no necessary sequential order between the above steps. Those skilled in the art can understand from the description of the embodiments of the present invention that the above steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in exchange, etc. Moreover, at least some steps in the embodiments or flowcharts may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but may be executed in turn, alternately, or synchronously with other steps or at least a part of the sub-steps or stages of other steps.

[0062] Based on the above method embodiments, please refer to Figure 7 Another embodiment of the present invention also provides a user emotion recognition device, wherein the device includes: The acquisition module 11 is used to acquire call recording data associated with the target user, parse the call recording data, and extract voice data and text data corresponding to the voice data from the parsing result. The speech framing module 12 is used to perform speech framing processing on the speech data, obtain the corresponding frequency domain features based on the speech frames obtained after speech framing, and extract features from the speech frame feature sequence composed of the frequency domain features corresponding to each speech frame through a preset speech emotion feature encoder to generate the speech features of the speech data. The text segmentation module 13 is used to perform text segmentation processing on the text data, obtain the corresponding text semantic features based on the text units obtained after text segmentation, and extract features from the text unit feature sequence composed of the text semantic features corresponding to each text unit through a preset text sentiment encoder to generate the text features of the text data. The recognition module 14 is used to generate fusion features of the call recording data based on the voice features and the text features, and to identify the emotion classification result of the target user based on the fusion features.

[0063] Furthermore, in one embodiment, the user emotion recognition device, wherein acquiring call recording data associated with the target user, parsing the call recording data, and extracting voice data and corresponding text data from the parsing result specifically includes: Obtain call recording data associated with the target user, perform validity verification on the call recording data, remove invalid recording segments with signal amplitude below a preset signal threshold, and obtain verified call recording data; The verified call recording data is subjected to audio parsing and separation processing to obtain the original audio stream. The original audio stream is then subjected to noise suppression processing to obtain noise-reduced voice data. The denoised speech data is transcribed to generate transcribed text, and a timestamp mapping relationship is established between the transcribed text and the denoised speech data to obtain text data that is time-aligned with the denoised speech data.

[0064] Further, in one embodiment, the user emotion recognition device, wherein performing speech frame segmentation on the speech data, obtaining corresponding frequency domain features based on the speech frames obtained after speech frame segmentation, and extracting features from the speech frame feature sequence composed of the frequency domain features corresponding to each speech frame through a preset speech emotion feature encoder to generate speech features of the speech data, specifically includes: The speech data is segmented into frames according to a preset frame length to obtain multiple speech frames. Each speech frame is then filtered for validity according to a preset frame quality threshold. Invalid speech frames with a noise ratio exceeding the preset frame quality threshold are removed, and valid speech frames are retained. Windowing and short-time Fourier transform are performed on each of the effective speech frames in sequence to obtain the corresponding frame spectrum. The frame spectrum is then mapped using a Mel filter bank to obtain the Mel spectrum. The Mel spectra corresponding to each of the effective speech frames are then concatenated to obtain the Mel spectrum sequence. The speech features of the speech data are generated by extracting features from the Mel spectrum sequence using a preset speech emotion feature encoder.

[0065] Further, in the aforementioned user emotion recognition device, the step of sequentially performing windowing processing and short-time Fourier transform on each of the effective speech frames to obtain the corresponding frame spectrum, mapping the frame spectrum using a Mel filter bank to obtain a Mel spectrum, and concatenating the Mel spectra corresponding to each of the effective speech frames to obtain a Mel spectrum sequence specifically includes: Windowing is applied to each of the effective speech frames, and a short-time Fourier transform is performed on each of the windowed effective speech frames to obtain the corresponding frame spectrum. Then, the amplitude of each frame spectrum is normalized according to a preset amplitude range to obtain the normalized frame spectrum. The normalized frame spectra are mapped using a Mel filter bank to obtain the Mel spectrum corresponding to each effective speech frame. The corresponding Mel spectrum is then dynamically weighted at the frame level based on the frame quality score of each effective speech frame to obtain the weighted Mel spectrum. The weighted Mel spectra corresponding to each of the effective speech frames are concatenated to obtain a Mel spectrum sequence.

[0066] Further, in one embodiment, the user emotion recognition device, wherein the text data is processed by text segmentation, and corresponding text semantic features are obtained based on the text units obtained after text segmentation, and features are extracted from the text unit feature sequence composed of the text semantic features corresponding to each text unit through a preset text emotion feature encoder to generate text features of the text data, specifically includes: The text data is processed by text segmentation to obtain multiple text units, and each text unit is filtered for confidence based on a preset transcription confidence threshold to obtain a reliable text unit. Each trusted text unit is subjected to word encoding, sentence encoding, and position encoding to obtain corresponding text semantic features. A corresponding text unit semantic mask is generated based on the transcription confidence of each trusted text unit. A text unit feature sequence is constructed based on the text semantic features and the text unit semantic mask. By using a preset text sentiment feature encoder, feature extraction is performed on the feature sequence of the text units to generate the text features of the text data.

[0067] Furthermore, in the aforementioned user emotion recognition device, the step of performing text segmentation processing on the text data to obtain multiple text units, and filtering the credibility of each text unit according to a preset transcription confidence threshold to obtain credible text units, specifically includes: The text data is segmented using a preset word segmenter, and special symbols and punctuation marks in the text data are removed to obtain multiple sub-word units after splitting. The multiple sub-word units are arranged in the original chronological order of the text data to form a sub-word unit sequence. A classifier is added at the beginning of the sub-word unit sequence and a separator is added at the end to form a text unit sequence. Each element in the text unit sequence is taken as a text unit to obtain multiple text units. Obtain the transcription confidence level corresponding to each text unit, and perform confidence screening on each text unit according to the preset transcription confidence level threshold. Remove low-confidence text units with transcription confidence levels lower than the preset transcription confidence level threshold, and retain the reliable text units.

[0068] Furthermore, in one embodiment, the user emotion recognition device, wherein generating fusion features of the call recording data based on the voice features and the text features, and identifying the emotion classification result of the target user based on the fusion features, specifically includes: Cross-modal feature alignment calibration is performed on the speech features and the text features to obtain dimension-aligned target speech features and target text features; The frame quality scores corresponding to each valid speech frame in the speech data are aggregated and calculated to obtain the speech-side frame quality score. The transcription confidence scores corresponding to each reliable text unit in the text data are aggregated and calculated to obtain the text-side transcription confidence scores. Based on the target speech features, the target text features, the speech-side frame quality score, and the text-side transcription confidence, multimodal weighted fusion is performed to generate the fusion features of the call recording data; Based on the fused features, sentiment inference and recognition are performed to generate the sentiment classification result for the target user.

[0069] It should be noted that, in the device embodiments of the present invention, the information interaction and execution process between the above modules are based on the same concept as in the method embodiments of the present invention. For details on their specific functions and the resulting technical effects, please refer to the aforementioned method embodiments section, which will not be repeated here.

[0070] Based on the above method embodiments, another embodiment of the present invention also provides a computer device, which can be a server, and its internal structure diagram can be as follows. Figure 8 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements the functions or steps of the user emotion recognition method server-side as described in any of the above method embodiments.

[0071] Based on the above method embodiments, another embodiment of the present invention also provides a computer device, which can be a client, and its internal structure diagram can be as follows. Figure 9As shown, the computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements the functions or steps of the user emotion recognition method on the client side as described in any of the above method embodiments.

[0072] Those skilled in the art will understand that Figure 8 and Figure 9 The structural schematic diagram shown is only a schematic diagram of a part of the structure related to the present invention and does not constitute a limitation on the computer device on which the present invention is applied. The specific computer device may include more components than shown in the figure, or combine certain components, or have different component arrangements.

[0073] The processor referred to here can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0074] The memory includes readable storage media, internal memory, etc., where internal memory can be the RAM of a computer device. Internal memory provides an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of the computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory can include both internal storage units and external storage devices of the computer device. The memory is used to store the operating system, applications, boot loader, data, and other programs, such as program code for computer programs. The memory can also be used to temporarily store data that has been output or will be output.

[0075] Based on the above method embodiments, another embodiment of the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the user emotion recognition method as described in any of the above method embodiments. The computer-readable storage medium may be non-volatile or volatile.

[0076] It should be noted that the functions or steps that can be achieved by the computer-readable storage medium or computer device, and the technical effects brought about by the functions / steps, can be referred to the relevant descriptions in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0077] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc. The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0078] Those skilled in the art will understand that, for the sake of convenience and brevity, the embodiments of the device of the present invention are only illustrated by the division of the above-mentioned functional units and modules. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of the present invention. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0079] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0080] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0081] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0082] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0083] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A user emotion recognition method, characterized in that, include: Obtain call recording data associated with the target user, parse the call recording data, and extract voice data and text data corresponding to the voice data from the parsing results; The speech data is processed by speech framing. Based on the speech frames obtained by speech framing, the corresponding frequency domain features are obtained. Through a preset speech emotion feature encoder, the speech frame feature sequence composed of the frequency domain features corresponding to each speech frame is extracted to generate the speech features of the speech data. The text data is processed by text segmentation. Based on the text units obtained by text segmentation, the corresponding text semantic features are obtained. Through a preset text sentiment encoder, the text unit feature sequence composed of the text semantic features corresponding to each text unit is extracted to generate the text features of the text data. Based on the voice features and the text features, a fusion feature of the call recording data is generated, and based on the fusion feature, the emotion classification result of the target user is identified.

2. The user emotion recognition method according to claim 1, characterized in that, The process of acquiring call recording data associated with the target user, parsing the call recording data, and extracting voice data and corresponding text data from the parsing results includes: Obtain call recording data associated with the target user, perform validity verification on the call recording data, remove invalid recording segments with signal amplitude below a preset signal threshold, and obtain verified call recording data; The verified call recording data is subjected to audio parsing and separation processing to obtain the original audio stream. The original audio stream is then subjected to noise suppression processing to obtain noise-reduced voice data. The denoised speech data is transcribed to generate transcribed text, and a timestamp mapping relationship is established between the transcribed text and the denoised speech data to obtain text data that is time-aligned with the denoised speech data.

3. The user emotion recognition method according to claim 1, characterized in that, The process of performing speech framing on the speech data, obtaining corresponding frequency domain features based on the speech frames obtained from the speech framing, and extracting features from the speech frame feature sequence composed of the frequency domain features corresponding to each speech frame using a preset speech emotion feature encoder to generate speech features of the speech data, includes: The speech data is segmented into frames according to a preset frame length to obtain multiple speech frames. Each speech frame is then filtered for validity according to a preset frame quality threshold. Invalid speech frames with a noise ratio exceeding the preset frame quality threshold are removed, and valid speech frames are retained. Windowing and short-time Fourier transform are performed on each of the effective speech frames in sequence to obtain the corresponding frame spectrum. The frame spectrum is then mapped using a Mel filter bank to obtain the Mel spectrum. The Mel spectra corresponding to each of the effective speech frames are then concatenated to obtain the Mel spectrum sequence. The speech features of the speech data are generated by extracting features from the Mel spectrum sequence using a preset speech emotion feature encoder.

4. The user emotion recognition method according to claim 3, characterized in that, The process involves sequentially performing windowing and short-time Fourier transform on each of the effective speech frames to obtain the corresponding frame spectrum, mapping the frame spectrum using a Mel filter bank to obtain a Mel spectrum, and concatenating the Mel spectra corresponding to each of the effective speech frames to obtain a Mel spectrum sequence, including: Windowing is applied to each of the effective speech frames, and a short-time Fourier transform is performed on each of the windowed effective speech frames to obtain the corresponding frame spectrum. Then, the amplitude of each frame spectrum is normalized according to a preset amplitude range to obtain the normalized frame spectrum. The normalized frame spectra are mapped using a Mel filter bank to obtain the Mel spectrum corresponding to each effective speech frame. The corresponding Mel spectrum is then dynamically weighted at the frame level based on the frame quality score of each effective speech frame to obtain the weighted Mel spectrum. The weighted Mel spectra corresponding to each of the effective speech frames are concatenated to obtain a Mel spectrum sequence.

5. The user emotion recognition method according to claim 1, characterized in that, The text data is segmented, and the corresponding semantic features of the segmented text units are obtained. A preset text sentiment encoder is used to extract features from the text unit feature sequence composed of the semantic features corresponding to each text unit, generating the text features of the text data. This process includes: The text data is processed by text segmentation to obtain multiple text units, and each text unit is filtered for confidence based on a preset transcription confidence threshold to obtain a reliable text unit. Each trusted text unit is subjected to word encoding, sentence encoding, and position encoding to obtain corresponding text semantic features. A corresponding text unit semantic mask is generated based on the transcription confidence of each trusted text unit. A text unit feature sequence is constructed based on the text semantic features and the text unit semantic mask. By using a preset text sentiment feature encoder, feature extraction is performed on the feature sequence of the text units to generate the text features of the text data.

6. The user emotion recognition method according to claim 5, characterized in that, The process of segmenting the text data to obtain multiple text units, and then filtering the confidence of each text unit according to a preset transcription confidence threshold to obtain reliable text units, includes: The text data is segmented using a preset word segmenter, and special symbols and punctuation marks in the text data are removed to obtain multiple sub-word units after splitting. The multiple sub-word units are arranged in the original chronological order of the text data to form a sub-word unit sequence. A classifier is added at the beginning of the sub-word unit sequence and a separator is added at the end to form a text unit sequence. Each element in the text unit sequence is taken as a text unit to obtain multiple text units. Obtain the transcription confidence level corresponding to each text unit, and perform confidence screening on each text unit according to the preset transcription confidence level threshold. Remove low-confidence text units with transcription confidence levels lower than the preset transcription confidence level threshold, and retain the reliable text units.

7. The user emotion recognition method according to claim 1, characterized in that, The process of generating fused features of the call recording data based on the voice features and the text features, and identifying the emotion classification result of the target user based on the fused features, includes: Cross-modal feature alignment calibration is performed on the speech features and the text features to obtain dimension-aligned target speech features and target text features; The frame quality scores corresponding to each valid speech frame in the speech data are aggregated and calculated to obtain the speech-side frame quality score. The transcription confidence scores corresponding to each reliable text unit in the text data are aggregated and calculated to obtain the text-side transcription confidence scores. Based on the target speech features, the target text features, the speech-side frame quality score, and the text-side transcription confidence, multimodal weighted fusion is performed to generate the fusion features of the call recording data; Based on the fusion features, sentiment inference and recognition are performed to generate the sentiment classification result for the target user.

8. A user emotion recognition device, characterized in that, include: The acquisition module is used to acquire call recording data associated with the target user, parse the call recording data, and extract voice data and text data corresponding to the voice data from the parsing results. The speech framing module is used to perform speech framing processing on the speech data, obtain the corresponding frequency domain features based on the speech frames obtained by speech framing, and extract features from the speech frame feature sequence composed of the frequency domain features corresponding to each speech frame through a preset speech emotion feature encoder to generate the speech features of the speech data. The text segmentation module is used to perform text segmentation processing on the text data, obtain the corresponding text semantic features based on the text units obtained from the text segmentation, and extract features from the text unit feature sequence composed of the text semantic features corresponding to each text unit through a preset text sentiment feature encoder to generate the text features of the text data. The recognition module is used to generate fusion features of the call recording data based on the voice features and the text features, and to identify the emotion classification result of the target user based on the fusion features.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the user emotion recognition method as described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the user emotion recognition method as described in any one of claims 1-7.

Images