Service content generation method and device based on emotion recognition, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence and the field of financial science and technology, and discloses a service content generation method based on emotion recognition, and the method comprises the steps: collecting voice data, extracting voiceprint features, and building a user voiceprint file; extracting Mel frequency cepstrum coefficient features and Mel energy difference coefficient features, and inputting the extracted features into an emotion classification model for emotion classification recognition to obtain an emotion state of the user; the current emotional state is compared with a historical normal emotional state, emotional fluctuation is analyzed, and whether the emotional state of the user is abnormal or not is judged; and when it is detected that the emotional state is abnormal, target service content is determined from the personalized service library in combination with the emotional state and the user voiceprint file. According to the method, the voice features are extracted and the user voiceprint file is established, so that the emotion fluctuation of the user can be accurately analyzed, whether abnormity exists or not is judged, accurate recommendation service is provided in combination with the personalized features and historical behavior data of the user, and the intelligent and personalized degree of vehicle-mounted voice interaction is improved.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and financial technology, and in particular to a service content generation method, device, equipment and storage medium based on emotion recognition. Background Art

[0002] In the in-vehicle scenario, the driver's hands and eyes are occupied by driving tasks, which puts higher requirements on the interaction mode. In order to improve driving safety and reduce the interference of visual and manual operations, voice interaction has gradually become an important function of the intelligent cockpit. However, the existing in-vehicle voice interaction technology still has many shortcomings in emotion recognition and personalized recommendations, and it is difficult to meet users' needs for intelligent and personalized services.

[0003] Existing in-vehicle voice interaction systems mainly rely on simple voice feature extraction for emotion recognition, which makes it difficult to accurately capture and analyze the user's emotional state. In the in-vehicle environment, due to the influence of noise and signal interference, emotion recognition models that rely only on a single feature often cannot effectively distinguish the user's emotional changes in different situations, resulting in low analysis accuracy of emotional fluctuations and prone to misjudgment or omission. This inaccurate recognition will affect the system's responsiveness in terms of service recommendations and safety prompts.

[0004] When existing in-vehicle systems detect changes in a user's emotional state, it is difficult to associate the emotional state with personalized services. Due to the lack of the ability to dynamically analyze the user's personalized characteristics and historical behavior, the service content recommended by the system when the user's emotions fluctuate often fails to meet the user's actual needs. This technical limitation of lacking association analysis makes the recommended content mismatched with the user's actual needs, reducing the personalization and accuracy of the user experience.

[0005] In financial and insurance scenarios, users also face similar emotional fluctuations and personalized needs during decision-making and consultation. When users are nervous or anxious, they may tend to choose low-risk products or show resistance to complex information, which places higher demands on the accuracy of personalized services. However, existing financial voice interaction systems also generally rely on simple emotion recognition technology, which makes it difficult to accurately capture and analyze user emotional changes. In the interactive environment of finance and insurance, noise, information density and emotional changes work together, resulting in emotion recognition models that rely only on a single feature being unable to distinguish users' subtle emotional fluctuations, affecting the accuracy of system recommendations.

[0006] When existing financial systems detect changes in a user's emotional state, they often fail to associate the emotional state with the user's personalized preferences and historical behaviors, resulting in a mismatch between the recommended service content and user needs. In addition, the financial system also lacks accurate criteria for judging abnormal user emotions. The identification of abnormal emotional states mostly relies on a single emotional state value and lacks comparative analysis with historical emotional states. This technical limitation makes it difficult for the system to respond promptly and accurately when emotions fluctuate, which not only affects the consistency of the user experience, but may also mislead financial decisions, reducing user trust and service effectiveness. Summary of the invention

[0007] The main purpose of the present invention is to provide a service content generation method, device, equipment and storage medium based on emotion recognition, aiming to solve the technical problem that the existing technology is difficult to provide personalized service content through accurate emotion recognition and emotional state analysis in vehicle-mounted scenarios.

[0008] To achieve the above object, the present invention provides a method for generating service content based on emotion recognition, comprising:

[0009] Collecting voice data and extracting voiceprint features from the voice data;

[0010] Establishing a user voiceprint file according to the voiceprint characteristics;

[0011] Extracting Mel frequency cepstral coefficient features and Mel energy difference coefficient features from the speech data;

[0012] Inputting the Mel frequency cepstral coefficient feature and the Mel energy difference coefficient feature into an emotion classification model, performing emotion classification and recognition through the emotion classification model, and obtaining the user's emotional state;

[0013] Compare the user's emotional state with historical normal emotional states, analyze the user's emotional fluctuations, and determine whether the user has an abnormal emotional state;

[0014] When it is detected that the user has an abnormal emotional state, the target service content is determined from the personalized service library according to the emotional state and the user voiceprint file.

[0015] Furthermore, to achieve the above-mentioned purpose, the present invention provides a service content generation device based on emotion recognition, comprising:

[0016] A voiceprint feature extraction module, used to collect voice data and extract voiceprint features from the voice data;

[0017] A voiceprint file management module, used to establish a user voiceprint file according to the voiceprint characteristics;

[0018] An emotion feature extraction module, used to extract Mel frequency cepstral coefficient features and Mel energy difference coefficient features from the speech data;

[0019] An emotion recognition and classification module, used to input the Mel frequency cepstral coefficient feature and the Mel energy difference coefficient feature into an emotion classification model, perform emotion classification and recognition through the emotion classification model, and obtain the user's emotional state;

[0020] An emotion fluctuation analysis module is used to compare the emotion state of the user with the historical normal emotion state, analyze the emotion fluctuation of the user, and determine whether the user has an abnormal emotion state;

[0021] The personalized service generation module is used to determine the target service content from the personalized service library according to the emotional state and the user voiceprint file when it is detected that the user has an abnormal emotional state.

[0022] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a service content generation program based on emotion recognition stored in the memory and executable on the processor, wherein the service content generation program based on emotion recognition, when executed by the processor, implements the steps of the service content generation method based on emotion recognition as described above.

[0023] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a service content generation program based on emotion recognition is stored, and when the service content generation program based on emotion recognition is executed by a processor, the steps of the service content generation method based on emotion recognition as described above are implemented.

[0024] Beneficial effects: The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses a service content generation method based on emotion recognition, which collects voice data, extracts voiceprint features, and establishes a user voiceprint file; extracts Mel frequency cepstral coefficient features and Mel energy differential coefficient features, and inputs the extracted features into an emotion classification model for emotion classification and recognition to obtain the user's emotional state; compares the current emotional state with the historical normal emotional state, analyzes emotional fluctuations, and determines whether the user has an abnormal emotional state; when an abnormal emotional state is detected, the target service content is determined from a personalized service library in combination with the emotional state and the user voiceprint file. The present invention extracts voice features and establishes a user voiceprint file, and uses an emotion classification model to accurately identify the user's emotional state, can accurately analyze the user's emotional fluctuations and determine whether there is an abnormality, and provides accurate recommendation services in combination with the user's personalized features and historical behavior data, thereby improving the intelligence and personalization of in-vehicle voice interaction and improving the satisfaction and safety of user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0026] Figure 1 A schematic diagram of an application environment of a method for generating service content based on emotion recognition in an embodiment of the present invention;

[0027] Figure 2 It is a flow chart of an embodiment of a method for generating service content based on emotion recognition according to the present invention;

[0028] Figure 3 It is a functional module diagram of a preferred embodiment of the service content generation device based on emotion recognition of the present invention;

[0029] Figure 4 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0030] Figure 5 FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0031] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0032] The method for generating service content based on emotion recognition provided by the embodiment of the present invention can be applied in the following aspects: Figure 1 In the application environment, the user terminal communicates with the server terminal through the network. The server terminal can collect voice data through the user terminal, extract voiceprint features, and establish a user voiceprint file; extract Mel frequency cepstral coefficient features and Mel energy difference coefficient features, input the extracted features into the emotion classification model for emotion classification and recognition, and obtain the user's emotional state; compare the current emotional state with the historical normal emotional state, analyze the emotional fluctuations, and determine whether the user has an abnormal emotional state; when the emotional state is detected to be abnormal, combine the emotional state and the user voiceprint file to determine the target service content from the personalized service library. The present invention extracts voice features and establishes a user voiceprint file, and uses the emotion classification model to accurately identify the user's emotional state, can accurately analyze the user's emotional fluctuations and determine whether there is an abnormality, and provides accurate recommendation services in combination with the user's personalized features and historical behavior data, thereby improving the intelligence and personalization of in-vehicle voice interaction, and improving the satisfaction and safety of user experience. Among them, the user terminal can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server terminal can be implemented by an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0033] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a method for generating service content based on emotion recognition provided by the present invention. It should be noted that although a logical sequence is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0034] like Figure 2 As shown, the service content generation method based on emotion recognition proposed by the present invention includes the following steps:

[0035] S10, collecting voice data, and extracting voiceprint features from the voice data;

[0036] In this embodiment, the collection of voice data refers to the real-time capture of the voice information of the driver and passengers in the vehicle through the microphone array device in the vehicle system. In the vehicle environment, the sources of noise are diverse and complex, including background music, air conditioning sound, passenger conversations, etc. in the car, as well as external road noise, wind noise and noise from other vehicles. Therefore, the collection of voice data needs to rely on microphone array technology to improve the signal-to-noise ratio of voice collection. The microphone array uses multiple microphones to work together, with the help of beamforming, noise suppression, echo cancellation and other algorithms to reduce the interference of environmental noise.

[0037] The vehicle system is equipped with multiple microphones, which are arranged in an array at appropriate locations in the vehicle (such as the steering wheel, the head of the driver's seat, and above the dashboard) to ensure that the microphone's collection range can cover the driver and passengers in the vehicle. The collected voice signals are processed using beamforming technology. Beamforming can use the relative positions between microphones and the time delay differences of voice signals to strengthen voice signals from a specific direction and suppress noise from other directions. The voice signals are further processed for noise suppression and echo cancellation. Noise suppression uses an adaptive filtering algorithm to improve voice clarity by identifying and weakening noise frequency bands; echo cancellation uses an adaptive echo suppression algorithm to eliminate sound echoes generated by the vehicle multimedia system.

[0038] Voiceprint feature extraction is to extract the core feature parameters that can reflect the speaker's characteristics by performing signal processing and feature analysis on the collected voice data. Voiceprint features contain unique information such as the speaker's physiological characteristics and vocalization methods, which are expressed in the voice data through frequency features, time domain features, and intonation features.

[0039] In addition to the basic frequency, time domain and timbre characteristics, voiceprint features can also include the following aspects:

[0040] Age: Changes in age will lead to changes in the vibration frequency and pitch of the vocal cords. These changes can be inferred by analyzing the spectral information, formant position, etc. of the voiceprint features. For example, the vocal cords of children and the elderly are different from those of middle-aged people, so the high-frequency components in the voice signal will be different. The user's age can be inferred through the changing trends of these features.

[0041] Gender: When men and women speak, there are significant differences in the frequency components and formant positions of speech signals due to physiological differences such as vocal tract length and vocal cord morphology. Men have longer vocal cords, and usually have lower formant positions, while women have shorter vocal cords and higher formant positions. Therefore, the gender of the user can be accurately distinguished through formant and frequency characteristics.

[0042] Accent and language habits: The user's voice signal also contains accent characteristics. Users from different regions or cultural backgrounds have different pronunciation methods, tones, and rhythms. For example, Mandarin and Cantonese have significant differences in speech frequency, syllable length, and tone changes. These accent characteristics can be captured through frequency domain analysis and tone feature extraction.

[0043] These physiological and social attributes of voiceprint characteristics play an important role in personalized services:

[0044] Personalized services based on age and gender: Users of different ages and genders have different needs and preferences for in-vehicle services. For example, the elderly may prefer to receive navigation prompts and simple in-vehicle operation instructions, while young users may prefer social and entertainment functions. By inferring the user's age and gender through voiceprint features, the system can provide more suitable personalized service content, such as adjusting the prompt volume or speech speed according to age, and recommending suitable content according to gender.

[0045] Accent-based language adaptation: By detecting the user's accent characteristics, the language setting of the in-vehicle voice system can be automatically switched. For example, when the system detects that the user speaks with a strong Cantonese accent, it can automatically switch to Cantonese broadcast or display Cantonese prompts, thereby improving the user's interactive experience.

[0046] Personalized recommendations based on social features: Voiceprint features can also be used to further explore users' interests and frequently used services through comprehensive analysis of historical user interaction data. For example, young users may like certain types of music or audio content, while older users may be more concerned about road safety tips and navigation content. The system can infer the user's basic information based on different voiceprint features, and then make intelligent recommendations based on the user's personalized profile.

[0047] The collected speech signal is preprocessed, including removing silent segments, normalizing the volume, filtering out low-frequency noise, etc. The preprocessed signal has better data quality. The preprocessed speech signal is feature extracted. Common feature extraction methods include: calculating the cepstral coefficients of the speech signal at the Mel frequency through the short-time Fourier transform (STFT) of the speech signal. Based on the linear predictive coding (LPC) algorithm, the cepstral coefficient features of the speech signal are calculated to capture the speaker's vocal characteristics. Through the human ear hearing model, linear prediction features that conform to human perception are extracted.

[0048] Different features (such as MFCC, LPCC, PLP) are integrated through normalization and multi-dimensional feature vectors to form a complete voiceprint feature data set. These feature data sets can be used for subsequent user identification or personalized recommendation services.

[0049] Example description: In the application scenario of the smart cockpit, the driver's hands and eyes are mostly occupied by driving tasks, making it difficult to perform manual operations. In order to achieve convenient voice control, high-quality voice acquisition and recognition capabilities are required. Through a multi-microphone array and advanced noise suppression algorithms, voice information can be efficiently collected in the noisy environment inside the car, and unique voiceprint features can be extracted to ensure that the system can accurately recognize the user's voice commands without interfering with driving.

[0050] Through efficient voice collection and multi-dimensional voiceprint feature extraction, it is possible to accurately collect user voice information in complex vehicle environments and extract stable and reliable voiceprint feature data. It can provide high-quality basic data for subsequent user identity recognition, emotion analysis and personalized services, thereby improving the accuracy of the entire system and user experience.

[0051] S20, establishing a user voiceprint file according to the voiceprint feature;

[0052] In this embodiment, the establishment of a user voiceprint profile is based on the extracted voiceprint features, and unique personalized information is recorded and modeled for each user. The user voiceprint profile includes not only the speaker's voiceprint features, but also the user's personalized preferences, voice behavior characteristics and other information. The voiceprint profile is used for subsequent identity recognition, emotion recognition, and personalized service recommendation functions.

[0053] The voiceprint feature data collected multiple times are clustered and analyzed to identify the user's common voice patterns. Through clustering algorithms (such as K-means, hierarchical clustering, etc.), the user's voiceprint features can be grouped to summarize the user's typical voice features. Based on the user's typical voiceprint features, the voiceprint features are modeled using technologies such as Gaussian mixture model (GMM), support vector machine (SVM) or deep neural network (DNN). This modeling process requires collecting user voices multiple times to train a model that can accurately identify user features. In addition to basic voiceprint features, the user's personalized preferences and commonly used command information can also be recorded. This information includes data such as the user's idioms, commonly used voice commands (such as navigation destinations, music preferences, etc.) and the user's interaction history. By recording this data, user needs can be better identified in subsequent interactions.

[0054] In order to improve the accuracy and timeliness of user voiceprint files, the system should update the voiceprint feature model based on the newly collected data after each user voice interaction. The sliding window or incremental learning method can be used to optimize the model in real time to ensure the accuracy and timeliness of user files.

[0055] Example description: In the actual application of the smart cockpit, the driver's voice characteristics may change with the driving situation. Through continuous voice data collection and voiceprint file updates, the user profile can be dynamically adjusted and optimized, so that the vehicle system can more accurately identify the voice commands of different users. In addition, the personalized information contained in the user's voiceprint file can also help the system quickly understand the user's needs, such as automatically recommending commonly used navigation routes, music playback and other personalized services, thereby improving user experience and driving safety.

[0056] By establishing a user voiceprint profile based on voiceprint features, the user's voice features and personalized preference information can be continuously recorded and updated. This can not only achieve efficient user identity recognition, but also provide support in personalized recommendations and sentiment analysis, thereby improving the intelligence level of the in-vehicle system.

[0057] S30, extracting Mel frequency cepstral coefficient features and Mel energy difference coefficient features from the speech data;

[0058] In this embodiment, Mel Frequency Cepstral Coefficient (MFCC) and Mel Energy Difference Coefficient (MEDC) are two types of features commonly used in speech signal processing. MFCC features extract the spectral information of speech signals by imitating the way the human auditory system perceives frequencies. This feature can effectively capture the core spectral characteristics of speech signals and is widely used in speech recognition and emotion recognition. MEDC features are obtained by differential calculation of MFCC features and are mainly used to capture the dynamic changes of speech signals. MEDC can reflect the instantaneous rate of change and acceleration information of speech signals, which is very useful for detecting changes in emotions and dynamic characteristics of sounds.

[0059] The collected speech signal is preprocessed, mainly including removing silence, denoising, normalizing volume, etc. This step is to improve the accuracy of feature extraction and the quality of data. The speech signal is divided into multiple overlapping short-time frames (such as 20-40 milliseconds per frame), and a windowing function such as a Hamming window or a Hanning window is applied to each frame. The purpose of framing is to obtain stable speech features in the frequency domain, and windowing can reduce spectral leakage.

[0060] Perform a Fast Fourier Transform on each frame to convert the time domain signal into the frequency domain and obtain the spectrum information of each frame. The result of FFT provides the energy distribution of the speech signal in the frequency domain.

[0061] The spectral information is converted into a Mel frequency scale that matches the hearing characteristics of the human ear through a set of designed Mel filters. The design of Mel filters is based on the fact that the human ear is more sensitive to low frequencies and has a lower resolution for high frequencies.

[0062] The spectral energy after Mel filtering is logarithmically operated (logarithm is taken). This step is to convert the multiplicative relationship in the speech signal into an additive relationship.

[0063] Perform discrete cosine transform on the logarithmic Mel spectrum to obtain a set of cepstral coefficients, which are Mel frequency cepstral coefficients (MFCC). The purpose of DCT is to extract the low-frequency components in the spectrum information and eliminate the correlation in the speech signal.

[0064] Based on the MFCC features, the first-order difference of the MFCC coefficients of each frame (i.e., the rate of change between adjacent frames) is calculated to obtain the first-order difference features (usually denoted as ΔMFCC). The first-order difference represents the rate of change of the MFCC features.

[0065] The rate of change of the first-order difference of MFCC is further calculated, that is, the difference of the first-order difference (usually recorded as ΔΔMFCC), to obtain the second-order difference feature. The second-order difference represents the acceleration of the change of the MFCC feature.

[0066] According to the first-order and second-order differences of MFCC, the energy changes between frames are calculated to obtain the energy difference features between frames. These energy change features are further combined with the first-order and second-order difference results of MFCC to form the Mel Energy Difference Coefficient (MEDC) features.

[0067] Example description: In the smart cockpit, the driver's voice signal may be affected by emotions and external environment. For example, when the driver is angry or anxious, the energy and spectrum distribution of the voice signal will change significantly. By extracting MFCC features, the spectral characteristics of these changes can be captured, and by extracting MEDC features, the dynamic characteristics of emotional changes can be further reflected. Combining these features for emotion recognition can provide the driver with appropriate reminders or adjust the vehicle environment when the driver is in a bad emotional state, thereby improving driving safety and comfort.

[0068] By extracting MFCC features and MEDC features, the spectrum information and dynamic change characteristics of the speech signal can be obtained. This combination can effectively capture the emotional information in the speech signal and improve the accuracy of emotion classification.

[0069] S40, inputting the Mel frequency cepstral coefficient feature and the Mel energy difference coefficient feature into an emotion classification model, performing emotion classification and recognition through the emotion classification model, and obtaining the user's emotional state;

[0070] In this embodiment, the previously extracted Mel Frequency Cepstral Coefficient (MFCC) features and Mel Energy Difference Coefficient (MEDC) features are used as input data of the emotion classification model to perform emotion classification and recognition. The emotion classification model determines the user's current emotional state, such as happiness, anger, anxiety, relaxation, etc., by analyzing these features.

[0071] Before inputting MFCC features and MEDC features into the emotion classification model, these feature data need to be standardized or normalized. The standardization and normalization of feature data is to eliminate the dimensional differences between data and avoid features with a large numerical range from having too much impact on the emotion classification results.

[0072] By subtracting the mean of the feature and dividing it by the standard deviation of the feature, the distribution of the feature data is made more consistent with the requirements of the normal distribution.

[0073] Scale feature data to a fixed range (e.g., between 0 and 1) to improve the stability and accuracy of model training.

[0074] Emotion classification models can use a variety of machine learning or deep learning methods, such as support vector machine (SVM), random forest, multi-layer perceptron (MLP) or convolutional neural network (CNN). Each method has its advantages and applicable scenarios. You can choose a suitable model based on the amount of data, feature complexity and emotion classification requirements. Including:

[0075] Support Vector Machine (SVM): It divides different emotion categories by constructing a hyperplane, which is suitable for small-scale data sets and linearly separable problems.

[0076] Multilayer Perceptron (MLP): It uses neural networks to perform multi-layer nonlinear transformations, can handle complex feature spaces, and is suitable for large-scale data sets.

[0077] Convolutional Neural Network (CNN): It is good at processing time series data and image data, and can extract the spatiotemporal relationship between features through the convolution layer, thereby improving classification accuracy.

[0078] The emotion classification model needs to be trained with a large amount of speech emotion data. During the training process, the feature vector is used as input, and the corresponding emotion label (such as happiness, anger, sadness, etc.) is used as output to train the model parameters. The trained emotion classification model should be able to accurately identify the emotion category corresponding to the input feature.

[0079] Collect diverse speech emotion data and annotate each sample with emotion category labels. These labels can be obtained through emotion datasets (such as IEMOCAP, RAVDESS, etc.) or emotion collection in actual vehicle scenarios.

[0080] The emotion classification model is trained using the dataset, and the classification accuracy of the model is evaluated through cross-validation (such as K-fold cross-validation). The purpose of the validation is to ensure that the model can accurately identify various emotional states in practical applications.

[0081] The normalized feature vector is input into the trained emotion classification model, and the prediction result of the emotion state is obtained through the forward propagation calculation of the model. The emotion classification model will classify the input feature data and output the user's current emotional state.

[0082] Example description: In the scenario of the smart cockpit, after the system collects the driver's voice signal, it extracts MFCC and MEDC features and uses these features as the input of the emotion classification model. When the driver is emotional, such as angry or anxious, the spectral information and dynamic changes of the voice signal will change significantly. Through the recognition of the emotion classification model, the system can accurately judge the driver's emotional state and provide comforting prompts or safety reminders when necessary, thereby improving driving safety and user experience.

[0083] By inputting MFCC features and MEDC features into the emotion classification model for emotion classification and recognition, the user's emotional information in the voice can be effectively captured and the user's current emotional state can be accurately judged. The emotion classification model can detect the user's emotional state in real time based on the changes in the user's voice features, and provide appropriate personalized services or safety prompts when the emotion is abnormal.

[0084] S50, comparing the user's emotional state with a historical normal emotional state, analyzing the user's emotional fluctuations, and determining whether the user has an abnormal emotional state;

[0085] In this embodiment, the current user's emotional state is compared with the historical normal emotional state, the user's emotional fluctuations are analyzed, and it is determined whether the user has an abnormal emotional state. The purpose of this process is to establish a "normal emotional benchmark" through historical emotional data, and use this benchmark to evaluate whether the user's current emotional state deviates from the normal range, thereby determining whether the user has abnormal emotions (such as anxiety, anger, fatigue, etc.).

[0086] Based on the user's historical emotional data, the system needs to establish a "normal emotional benchmark". This benchmark can be used to extract the average state and fluctuation range of emotions through statistical analysis of the user's historical emotional state. The specific method is as follows:

[0087] Collection and storage of historical emotion data: During daily use, the system will continuously collect the user's emotional state data and store this data in the user's emotion profile. Emotion data includes information such as the user's different emotion categories (such as happiness, calmness, anger, etc.) and their corresponding intensity and duration.

[0088] Calculation of baseline state: Analyze historical emotional state data and calculate the average and standard deviation of emotional category, emotional intensity and emotional duration. This can establish a baseline range of emotional states. For example, a user's normal emotional baseline may be "emotional intensity of 5 (out of 10) in a calm state and duration of 10 minutes."

[0089] After the user's current emotional state is detected, it needs to be compared with the historical normal emotional baseline. This comparison can be done in multiple dimensions, including emotion category, emotion intensity, and emotion duration. Determine whether the user's current emotion category is consistent with their normal emotion category. For example, if the user is usually calm while driving, and the currently detected emotion category is "angry" or "anxious", there may be an abnormal emotion. Calculate the difference between the current emotion intensity and the historical emotion intensity. For example, if the user's normal emotion intensity baseline is 5, and the current detected intensity is 8, it means that the emotion intensity is significantly high.

[0090] Evaluate the duration of the current emotional state compared to the normal duration. If the user's normal emotional state usually lasts 10 minutes, and the current emotional state lasts significantly longer, there may be an emotional abnormality.

[0091] Based on the comparison results of emotion category, emotion intensity and duration, the system scores the user's emotion fluctuation. The emotion fluctuation score can be calculated based on the following formula:

[0092] Mood swing score = (emotion category difference × weight 1) + (emotion intensity difference × weight 2) + (emotion duration difference × weight 3)

[0093] Among them, each weight can be adjusted according to different application scenarios and emotional sensitivity. Finally, by comparing the emotional fluctuation score with the preset abnormal threshold, it is determined whether the user has an abnormal emotional state.

[0094] Example description: In customer service scenarios in the financial or insurance industries, personalized recommendations and risk assessments increasingly rely on real-time analysis of customer emotions and psychological states. For example, in an intelligent financial assistant or insurance advisor system, customers interact with the system through voice to express their insurance needs or financial consultations. Customers' emotional states may affect their decision-making process, so timely identification and analysis of customer emotions is of great significance.

[0095] Assume that a customer is consulting about insurance products and is communicating with the insurance advisor system through voice. The system collects and analyzes the voice to extract the customer's Mel Frequency Cepstral Coefficient (MFCC) features and Mel Energy Difference Coefficient (MEDC) features, and then inputs these features into the emotion classification model for emotion recognition. The system determines that the customer is in a state of anxiety or tension, which may be due to insufficient understanding of the complex insurance terms or concerns about the current financial market.

[0096] By comparing the customer's current emotional state with his or her historical normal emotional state, the system finds that the customer's emotional fluctuations are large, the emotional intensity is much higher than usual, and the duration is longer. This kind of emotional fluctuation is considered an abnormal state. Based on the emotional fluctuation score and the preset emotional abnormality threshold, the system determines that the customer's emotional state is abnormal and may affect his or her decision-making.

[0097] To this end, the system proactively adjusts the way of dialogue and provides more concise and easy-to-understand explanations of insurance products to help customers relieve their nervousness. For example, the system can re-explain insurance terms in simple and clear language, or recommend some more conservative and stable financial products. At the same time, the system will remind customers not to rush to make decisions, and suggest that they conduct more in-depth consultations later or arrange further communication with professional consultants.

[0098] In addition, the system can analyze the customer's preference for different financial products based on the customer's emotional fluctuation history. For example, when the customer is in a state of high anxiety, he or she tends to choose low-risk products. The system can store these emotional and behavioral data in the customer's personalized profile to better optimize the personalized recommendation strategy in the future.

[0099] By comparing the current emotional state with the historical normal emotional state, the user's emotional fluctuations and abnormalities can be accurately detected. Comprehensive analysis based on multi-dimensional emotional data can be performed to avoid misjudgment caused by judging based on a single emotional indicator.

[0100] S60, when it is detected that the user has an abnormal emotional state, determine target service content from a personalized service library according to the emotional state and the user voiceprint file.

[0101] In this embodiment, when the user's emotional state is detected to be abnormal, it is necessary to select the most suitable service content from the personalized service library based on the user's current emotional state information and the personalized data in his voiceprint file. The abnormal emotional state may mean that the user is in a state of anger, anxiety, fatigue, etc., and the voiceprint file records the user's preferences, habits, language style, etc. The personalized service library contains various service options, such as soothing reminders, playing specific music, providing navigation suggestions, etc.

[0102] When it is detected that the user's emotional state is abnormal, it is first necessary to filter out candidate service content related to the current emotional state from the personalized service library. The types of abnormal emotional states can be anxiety, anger, sadness, etc. Different emotional types correspond to different candidate service content. For example, when the user's emotional state is "anxiety", candidate service content such as "soothing reminder" and "playing relaxing music" can be filtered out; when the user's emotional state is "anger", candidate service content such as "calming suggestions" and "playing soothing music" can be filtered out.

[0103] Extract the user's personalized features from the user's voiceprint file. These features include the user's basic information (such as age, gender, etc.), habitual behavior (such as favorite music types, common navigation paths, etc.) and historical preference data (such as the service content received in the past under similar emotional states). Through these personalized features, we can better understand the user's preferences and needs, and improve the accuracy and personalization of recommended services.

[0104] Based on the extracted personalized features, the candidate service content is further screened and matched to select the target service content that best meets the user's needs. The matching process can be based on similarity calculation, weighted scoring, etc. For example, by analyzing the user's historical behavior data, the similarity score of each candidate service content is calculated; combined with the user's personalized features (such as age, gender, preferences, etc.), weighted to obtain the final comprehensive score. The candidate content with the highest comprehensive score is the target service content.

[0105] Example: In the intelligent financial advisor system, customers interact with the system through voice to consult the characteristics, risks and benefits of different financial products. The system not only responds to the customer's voice instructions, but also monitors the customer's emotional state in real time to provide personalized services that are more in line with the customer's current needs.

[0106] During the voice interaction process, the system uses voice acquisition technology to extract the customer's voice features, including Mel Frequency Cepstral Coefficients (MFCC) and Mel Energy Difference Coefficients (MEDC). These features are input into the emotion classification model to identify the customer's emotional state, such as anxiety, doubt, or excitement. The system compares the customer's current emotional state with his or her historical normal emotional state, analyzes the emotion category, intensity, and duration, and generates an emotional fluctuation score. If the emotional fluctuation score exceeds the preset threshold, the system determines that the customer has an abnormal emotional state.

[0107] After the system detects the customer's anxiety, it will select suitable service content from the personalized service library based on the customer's emotional state and personalized profile. For example, when a customer shows anxiety, the system will give priority to recommending low-risk financial products to help the customer relieve anxiety and simplify the explanation of high-risk investment products. The system will extract personalized features (such as risk preference, historical product selection preference, etc.) from the customer's voiceprint profile, further screen candidate service content, and select the target service content that best meets the customer's needs.

[0108] After providing recommendation services, the system also collects customer feedback and operational behavior. For example, the system records the customer's acceptance of the recommended product (such as selection, rejection, consultation for more details, etc.), and whether the customer further asks for other investment advice. The system associates this feedback information with the customer's emotional state and personalized profile to generate a user interaction dataset.

[0109] Based on the collected user interaction data, the system will optimize and update the parameters of the emotion classification model to improve the accuracy of identifying customer emotions. For example, if a customer repeatedly rejects high-risk product recommendations while in an anxious mood, the system will adjust the weight of the emotion model so that it will prioritize recommending low-risk products when similar emotions are detected. In addition, the system will also update the customer's voiceprint profile, such as recording the customer's service preferences and acceptance in different emotional states, so as to provide more accurate personalized services in the future.

[0110] When a customer consults again or has a big mood swing, the system can accurately identify the customer's emotional state and dynamically adjust the recommendation strategy. In this way, the system can not only improve customer satisfaction, but also reduce decision-making errors caused by emotional fluctuations. During the use process, customers not only feel the professionalism of the intelligent consultant, but also enjoy personalized financial services that match their emotions and preferences.

[0111] By screening and matching target service content from the personalized service library according to the user's emotional state and personalized characteristics, personalized services that meet user needs can be provided more accurately. This not only improves the user's service experience, but also provides appropriate intervention measures when the user's emotional state is abnormal, avoiding the negative impact of bad emotions on users.

[0112] The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses a service content generation method based on emotion recognition, which collects voice data, extracts voiceprint features, and establishes a user voiceprint file; extracts Mel frequency cepstral coefficient features and Mel energy difference coefficient features, and inputs the extracted features into an emotion classification model for emotion classification and recognition to obtain the user's emotional state; compares the current emotional state with the historical normal emotional state, analyzes emotional fluctuations, and determines whether the user has an abnormal emotional state; when an abnormal emotional state is detected, the target service content is determined from a personalized service library in combination with the emotional state and the user voiceprint file. The present invention extracts voice features and establishes a user voiceprint file, and uses an emotion classification model to accurately identify the user's emotional state, can accurately analyze the user's emotional fluctuations and determine whether there is an abnormality, and provides accurate recommendation services in combination with the user's personalized features and historical behavior data, thereby improving the intelligence and personalization of in-vehicle voice interaction and improving the satisfaction and safety of user experience.

[0113] In one embodiment, the above S30 includes:

[0114] S301, performing fast Fourier transform on the speech data to obtain a spectrum signal;

[0115] S302, filtering the spectrum signal by using a Mel filter bank;

[0116] S303, performing logarithmic transformation on the filtered spectrum signal to obtain a logarithmic Mel spectrum;

[0117] S304, performing discrete cosine transform on the logarithmic Mel frequency spectrum to obtain the Mel frequency cepstral coefficient feature;

[0118] S305, analyzing the inter-frame first-order difference and the inter-frame second-order difference of the logarithmic Mel spectrum to obtain inter-frame first-order difference features and inter-frame second-order difference features respectively;

[0119] S306, smoothing the first-order difference feature between frames and the second-order difference feature between frames to obtain the Mel energy difference coefficient feature.

[0120] In this embodiment, Fast Fourier Transform (FFT) is a mathematical method for converting a speech signal in the time domain into a frequency domain signal. The speech signal contains the superposition of various frequency components in the time domain. These frequency components can be extracted using FFT to form the spectrum information of the speech signal. The spectrum signal can represent the frequency distribution and amplitude change of the speech signal. The system performs short-time frame processing (for example, 20ms to 40ms per frame) on the collected speech data, and performs Fast Fourier Transform (FFT) on the signal of each frame to obtain the spectrum of each frame. In this way, the system can extract the energy distribution of the speech signal at different frequencies.

[0121] Mel filter bank is a set of filters designed to mimic the auditory characteristics of the human ear. It provides higher resolution in the low frequency band and lower resolution in the high frequency band, which is consistent with the auditory characteristics of the human ear. Through Mel filter bank processing, frequency components that conform to human ear perception can be extracted more effectively. For each frame of the spectrum signal, a set of Mel filters (such as 24 filters) is applied to convert the spectrum information into frequency information that conforms to the Mel frequency scale. These filters are bandpass filters that cover different parts of the frequency range and obtain the energy of each frequency band through filtering.

[0122] Logarithmic transformation of the spectral signal after Mel filtering can convert the multiplicative relationship in the speech signal into an additive relationship and reduce the amplitude dynamic range. Logarithmic transformation can make the spectral data more consistent with the perceptual characteristics of the speech signal. Take the logarithm of the band energy value obtained by the Mel filter bank to obtain the logarithmic Mel spectrum. After logarithmic transformation of the energy of each band, the frequency characteristic changes of the speech signal can be more clearly reflected.

[0123] Discrete cosine transform (DCT) is a mathematical method that converts frequency domain signals into cepstrum signals. Through DCT, the spectral information in the logarithmic Mel spectrum can be compressed and de-correlated to obtain a set of cepstrum coefficients, which are the Mel-frequency cepstrum coefficients (MFCC). DCT operation is performed on the logarithmic Mel spectrum of each frame to extract the first several cepstrum coefficients (usually 12 to 13), which are the Mel-frequency cepstrum coefficient features. The purpose of DCT is to extract low-frequency features in speech signals, thereby eliminating the influence of high-frequency noise.

[0124] Inter-frame difference features are used to capture the dynamic changes of speech signals. The first-order difference features represent the rate of change of speech signals, and the second-order difference features represent the acceleration of the change. Inter-frame difference analysis can reveal the temporal variation characteristics of speech signals, which is of great significance for emotion recognition. The difference between the MFCC features of each frame is calculated to obtain the first-order difference features and the second-order difference features (i.e., ΔMFCC and ΔΔMFCC). These features reflect the dynamic changes and temporal fluctuation information of speech signals.

[0125] Smoothing is used to reduce the noise and jitter of the inter-frame differential features, making the features smoother and more stable. The smoothed features are called Mel Energy Differential Coefficient features (MEDC). Weighted moving average or Gaussian smoothing is applied to the first-order differential features and second-order differential features to reduce their fluctuations and jitter. The smoothed features are more stable and can better reflect the overall trend of the speech signal.

[0126] This embodiment can obtain stable spectrum information and dynamic change characteristics from the speech signal by extracting MFCC and MEDC features. The combination of these features can effectively capture the emotional information in the speech signal and provide accurate data support for subsequent emotion recognition. At the same time, the MFCC feature reflects the static spectrum characteristics of the speech signal, while the MEDC feature reflects the dynamic changes of the speech signal, which greatly improves the accuracy of emotion classification.

[0127] In one embodiment, in the above S60, when it is detected that the user has an abnormal emotional state, the process further includes:

[0128] S601, generating a safety prompt according to the emotional state, and recording feedback information on the safety prompt;

[0129] S602, adjusting the content of the security prompt according to the feedback information;

[0130] S603: Store the emotional state and the feedback information in the user voiceprint file.

[0131] In this embodiment, when an abnormal emotional state of the user is detected (such as anger, anxiety or fatigue, etc.), the system will generate a safety prompt for the emotional state. The purpose of the safety prompt is to make the user aware of his or her emotional state so that appropriate actions can be taken (such as calming down, taking a break, or changing driving behavior). In addition, the system will record the user's feedback on the safety prompt, such as whether the prompt is adopted, whether the user further interacts or ignores the prompt, etc. Based on the identified emotional state, appropriate prompt content is generated. For example, when the user is in an angry state, a voice prompt of "Please drive calmly" can be played; when the user is tired, the user can be advised to take a short break. Monitor the user's behavior or voice feedback after receiving the prompt. Feedback information may include the user's action response (such as braking, slowing down, etc.) or voice commands (such as "turn off the prompt" or "understood").

[0132] Based on user feedback, the system can dynamically adjust subsequent safety prompts. The purpose of the adjustment is to better match the user's preferences and current emotional state, and to avoid repeated or invalid prompts. For example, if a user repeatedly rejects or ignores a prompt, the system can gradually reduce the frequency of similar prompts, or change the way the prompt is expressed. Through data analysis and pattern recognition, analyze the user's feedback behavior and determine the user's acceptance and response to the prompt. Based on the analysis results, adjust the content, frequency and expression of the prompt. For example, a simple voice prompt can be adjusted to a voice content with an explanation of the reason, or a simple instruction can be added to the visual prompt.

[0133] The user's emotional state information and feedback information on safety tips should be stored in the user's voiceprint file. The recording of emotional state and feedback information helps to establish a more comprehensive user portrait and provide basic data for subsequent personalized services. Record the emotional state detected each time, the corresponding time and background scene (such as driving conditions) in the user's voiceprint file. Record the user's feedback behavior on safety tips and associate it with the emotional state. Through long-term accumulated data, a behavioral profile of emotional patterns can be established for users.

[0134] This embodiment generates safety tips, records feedback information and makes dynamic adjustments after detecting abnormal emotional states, so that the system can provide users with more personalized tips and assistance in different emotional states. At the same time, storing emotional states and feedback information in user voiceprint files helps to understand users' emotional characteristics and reaction patterns in the long term, thereby improving the intelligence level of the vehicle system and the user's safety experience.

[0135] In one embodiment, the above S60 includes:

[0136] S604, when it is detected that the user has an abnormal emotional state, candidate service content related to the emotional state is screened out from the personalized service library according to the emotional state;

[0137] S605, extracting personalized features from the user voiceprint file;

[0138] S606: Determine the target service content that best meets the user's preference from the candidate service contents based on the personalized features.

[0139] In this embodiment, when it is detected that the user has an abnormal emotional state (such as anger, anxiety or fatigue, etc.), the system will filter out candidate service content related to the emotional state from the personalized service library according to the emotional state. Different emotional states require corresponding different types of service content to effectively alleviate or solve the user's emotional problems. For example, when the user's emotional state is "anxiety", the candidate service content may include playing relaxing music, providing deep breathing exercises or rest reminders, etc. Predefine different emotional states (such as anger, anxiety, fatigue, etc.) and corresponding candidate service content lists. This mapping rule can be designed through data analysis and user behavior patterns. Based on the detected emotional state, the system selects a set of candidate service content related to the emotional state from the personalized service library.

[0140] Extract personalized features from the user's voiceprint profile to further filter the target service content that best meets the user's needs. Personalized features include the user's basic information (such as age, gender, etc.), interest preferences, historical behavior data, acceptance of different services, etc. Access the user's voiceprint profile and extract personalized feature data from it. For example, the user's music preferences, preferred driving mode, commonly used navigation routes, etc. Use the extracted personalized features to compare with the feature requirements of the candidate service content, and calculate the adaptability for each candidate service content.

[0141] Based on the user's personalized characteristics, select the target service content that best matches the user's preferences from the candidate service content. This process involves scoring each candidate service content and selecting the content with the highest score as the final service content. Build a scoring model to match the candidate service content with the user's personalized characteristics. For example, by calculating the similarity score between each candidate content and the user's preferences, the acceptance score of historical behavior data, etc. Based on the scoring results, select the candidate service content with the highest score as the final target service content.

[0142] This embodiment detects abnormal emotional states of users and filters target service content from the service library according to the personalized characteristics of users. The system can quickly select and provide personalized services that meet the needs of users in the case of abnormal emotions. It can provide users with targeted services more accurately, improve user comfort and user experience, especially in terms of emotional management and safety reminders.

[0143] In one embodiment, the above S606 includes:

[0144] S6061, analyzing the candidate service content based on the user's historical behavior data, and generating a historical behavior score for each candidate service content;

[0145] S6062, analyzing the similarity between the candidate service content and the personalized features, and generating a similarity score for each candidate service content;

[0146] S6063, analyzing the satisfaction of the candidate service content according to the historical feedback data of the user, and generating a satisfaction score for each candidate service content;

[0147] S6064, performing weighted synthesis of the historical behavior score, similarity score and satisfaction score to obtain a comprehensive score for each candidate service content;

[0148] S6065: Use the candidate service content with the highest comprehensive score as the target service content that best meets the user's preference.

[0149] In this embodiment, the system analyzes the candidate service content based on the user's historical behavior data and generates a historical behavior score for each candidate content. Historical behavior data generally includes the user's past choices and reaction behaviors under similar emotional states. For example, when a user is in an "anxious" emotional state, he or she often chooses some soothing music. Such behavior data can provide a basis for the historical behavior score of the service content. The system continuously collects the user's operational behavior data, such as voice commands, music playback, whether prompts are received or not, etc. A scoring model based on historical behavior data is established, and a historical behavior score is assigned to each candidate content based on the user's service selection behavior under similar emotional states in the past.

[0150] The system compares the candidate service content with the user's personalized features, analyzes the similarity between them, and generates a similarity score for each candidate content. Personalized features can include the user's music preferences, daily usage habits, etc. Define matching rules between different personalized features and candidate service content. Use this rule to judge the suitability of each candidate content. Use an algorithm based on feature similarity (such as Euclidean distance, cosine similarity, etc.) to score each candidate service content and generate a similarity score.

[0151] The system calculates the satisfaction score of each candidate content based on the user's historical feedback data on each candidate content. The satisfaction score is mainly based on the user's feedback information in the past use, such as acceptance, interaction frequency, positive or negative evaluation, etc. Collect and analyze the user's feedback data after using each service content, such as positive evaluation, negative feedback, etc. Use sentiment analysis, statistical analysis and other methods to calculate a satisfaction score for each candidate content.

[0152] The system weights and combines the historical behavior score, similarity score, and satisfaction score of each candidate service content to obtain a comprehensive score. The purpose of weighting is to reflect the importance of each score, and the specific weight setting can be adjusted according to the scenario and user habits. Design a weighting formula to calculate the three scores according to specific weights. For example, comprehensive score = (historical behavior score * 0.4) + (similarity score * 0.3) + (satisfaction score * 0.3). The weights can be dynamically adjusted based on the results of long-term data analysis to ensure the accuracy of the score.

[0153] According to the comprehensive scoring results, the candidate service content with the highest score is selected as the final target service content. The candidate content with the highest comprehensive score is usually the content that best meets the user's current needs and preferences. According to the ranking results of the comprehensive score, the candidate service content with the highest score is selected as the final target service content. After the target content is confirmed within the system, the execution of the content is triggered.

[0154] Based on the user's historical behavior data, personalized characteristics and historical feedback, the system can accurately screen out target service content that matches the user's current emotional state and personalized preferences. It can not only make intelligent recommendations based on the user's emotional state, but also comprehensively consider the user's personalized characteristics and satisfaction, thereby providing the best service experience.

[0155] In one embodiment, the above S50 includes:

[0156] S501, extracting historical emotional state data of the user from the user voiceprint file, and generating baseline data of the normal emotional state according to the historical emotional state data;

[0157] S502, analyzing the differences between the user's emotional state and a normal emotional state in terms of emotional category, emotional intensity, and emotional duration, respectively, based on the benchmark data;

[0158] S503, generating an emotion fluctuation score according to the differences in emotion category, emotion intensity and emotion duration, combined with the weight of each difference;

[0159] S504: judging whether the user has an abnormal emotional state according to the emotion fluctuation score and a preset abnormal fluctuation threshold.

[0160] In this embodiment, the system extracts the user's past emotional state data from the user's voiceprint archive, and these data reflect the user's emotional expression in different scenarios. Through statistical analysis of these data, the system can generate "normal emotional state baseline data" to represent the user's normal range in terms of different emotional categories, intensities and durations. The historical emotional state data output by the emotion recognition model is extracted from the user's voiceprint archive. These data include indicators such as emotional categories, intensity and duration. By performing statistical analysis such as mean and standard deviation on these historical emotional data, the normal emotional baseline state for each emotional category is determined. This baseline state can be expressed as a range value, such as the average intensity of a certain emotional category is 5 (out of 10 points), and the average duration is 10 minutes.

[0161] When the user's emotional state is detected, the system compares the emotional state with the normal emotional baseline data, and analyzes the differences in emotion category, emotion intensity, and emotion duration. This analysis helps determine whether the current emotion is within the normal range or whether there are obvious abnormalities. Check whether the category of the current emotional state is consistent or close to the category of the normal emotional baseline. If they are different, there may be a risk of emotional abnormality. Compare the intensity of the current emotional state with the mean and standard deviation of the intensity in the normal emotional baseline. If it is outside the normal range, it is considered to be abnormal in intensity. Analyze the difference between the duration of the current emotional state and the mean and standard deviation of the duration in the normal baseline. If the duration is too long or too short, there may also be emotional abnormality.

[0162] Based on the differences in emotion categories, intensity, and duration analyzed previously, the system assigns a certain weight to each difference and generates a mood swing score through weighted calculation. The mood swing score reflects the degree of deviation of the user's current emotional state from the normal baseline. The greater the deviation, the higher the score. Set a weight for each emotion difference (category, intensity, duration). The weight setting can be adjusted according to the degree of influence of the emotion on the user's behavior. Use a weighted formula to calculate the mood swing score, for example:

[0163] Mood swing score = (emotion category difference × weight 1) + (emotion intensity difference × weight 2) + (emotion duration difference × weight 3).

[0164] The values ​​of various weights can be dynamically adjusted based on actual user feedback and long-term data analysis to improve the accuracy of the volatility score.

[0165] Compare the calculated emotion fluctuation score with the preset abnormal fluctuation threshold. If the fluctuation score exceeds the threshold range, it is determined that the user's current emotion state is abnormal. The abnormal fluctuation threshold can be set according to the user's emotional characteristics and the security requirements of the system. According to data analysis and security policies, set a reasonable abnormal fluctuation threshold for each emotional state. Compare the emotion fluctuation score with the threshold to determine whether there is an emotion abnormality. If the emotion fluctuation score is higher than the threshold, the system will trigger the subsequent abnormal processing steps.

[0166] This embodiment can more accurately analyze the user's emotional fluctuations by comparing the user's current emotional state with the historical normal emotional state and comprehensively considering multi-dimensional factors such as emotional category, intensity and duration. By using the emotional fluctuation score and preset threshold for judgment, it is possible to detect the abnormal emotional state of the user in a timely manner, thereby triggering corresponding safety prompts or service adjustments. This improves the accuracy of emotional state monitoring and helps to effectively manage the user's emotional fluctuations in actual application scenarios.

[0167] In one embodiment, after the above S60, the method further includes:

[0168] S701, collecting feedback information and operation behaviors on the target service content;

[0169] S702, associating the feedback information and operation behavior with the corresponding emotional state and user voiceprint file to generate a user interaction data set;

[0170] S703, optimizing and updating parameters of the emotion classification model based on the user interaction dataset;

[0171] S704: Based on the user interaction data set, update the personalized features of the user voiceprint file.

[0172] In this embodiment, after the system determines and executes the target service content, it is necessary to collect user feedback information and operational behavior on the content. Feedback information may include the user's subjective evaluation (such as voice or touch screen feedback), and operational behavior may refer to the actions taken by the user after receiving the service (such as adjusting the driving mode, playing music, etc.). The purpose of this process is to obtain the user's true reaction to the current service content for further optimization of the recommendation model. Collect user feedback information through channels such as voice interaction, touch operation or system logs. Monitor the user's behavioral responses during the service execution process and record these behaviors as system logs.

[0173] The collected user feedback information and operation behavior are associated with the emotional state and voiceprint archive information to generate a complete user interaction data set. This data set contains information such as the user's emotional state, feedback behavior, operation log, etc., and is associated with the user's personalized characteristics. The feedback information, operation behavior, emotional state and voiceprint archive information are matched and associated. For example, the user's operation and feedback records in a specific emotional state are combined with the user's historical data. The associated information is organized into a unified data format and stored as a user interaction data set for subsequent data analysis and model optimization.

[0174] By analyzing the generated user interaction dataset, we can find out the user's acceptance of emotion classification and service content, and optimize the parameters of the emotion classification model accordingly. This process helps the emotion classification model to more accurately identify the user's emotional state and improve the overall performance of the model. Through the user behavior and feedback information in the interaction dataset, we can re-evaluate the accuracy of the emotion classification model, find out the shortcomings of the model, and update the model parameters in a targeted manner (such as adjusting weights or adding new features). Retrain the model with the new interaction dataset to improve the performance of the emotion classification model.

[0175] Based on the user's latest interactive behavior and feedback data, the personalized features in the user's voiceprint profile are updated. The user's voiceprint profile records information such as the user's preferences and emotional characteristics, which can be dynamically updated and optimized through long-term interactive behaviors. Analyze the user's behavior patterns in the interactive data set, extract the user's new preferences and emotional characteristics, and add them as new features to the user's voiceprint profile. According to the system's rules and algorithms, the content of the user's voiceprint profile is incrementally updated to more accurately reflect the user's needs in subsequent emotion recognition and personalized services.

[0176] This embodiment collects user feedback and operation behaviors after the target service content is executed, and associates them with emotional states and user profile information, so that the system can generate a comprehensive user interaction data set. Based on this data set, the emotion classification model can be dynamically optimized to improve the accuracy of emotion recognition. At the same time, by updating the personalized features in the user's voiceprint profile, the system can more flexibly provide personalized services that meet the user's latest preferences and improve the user experience.

[0177] In one embodiment, a service content generation device based on emotion recognition is provided, and the service content generation device based on emotion recognition corresponds one-to-one to the service content generation method based on emotion recognition in the above embodiment. Figure 3 , Figure 3This is a functional module diagram of a preferred embodiment of the service content generation device based on emotion recognition of the present invention. Voiceprint feature extraction module 10, voiceprint archive management module 20, emotion feature extraction module 30, emotion recognition and classification module 40, emotion fluctuation analysis module 50 and personalized service generation module 60. The detailed description of each functional module is as follows:

[0178] A voiceprint feature extraction module 10 is used to collect voice data and extract voiceprint features from the voice data;

[0179] A voiceprint file management module 20, used to establish a user voiceprint file according to the voiceprint features;

[0180] An emotion feature extraction module 30, used to extract Mel frequency cepstral coefficient features and Mel energy difference coefficient features from the speech data;

[0181] The emotion recognition and classification module 40 is used to input the Mel frequency cepstral coefficient feature and the Mel energy difference coefficient feature into an emotion classification model, perform emotion classification and recognition through the emotion classification model, and obtain the user's emotional state;

[0182] The emotion fluctuation analysis module 50 is used to compare the emotion state of the user with the historical normal emotion state, analyze the emotion fluctuation of the user, and determine whether the user has an abnormal emotion state;

[0183] The personalized service generation module 60 is used to determine target service content from the personalized service library according to the emotional state and the user voiceprint file when it is detected that the user has an abnormal emotional state.

[0184] In one embodiment, the emotion feature extraction module 30 is specifically used to:

[0185] Performing fast Fourier transform on the speech data to obtain a spectrum signal;

[0186] Performing filtering processing on the spectrum signal by using a Mel filter bank;

[0187] Perform logarithmic transformation on the filtered spectrum signal to obtain a logarithmic Mel spectrum;

[0188] Performing discrete cosine transform on the logarithmic Mel frequency spectrum to obtain the Mel frequency cepstral coefficient feature;

[0189] Analyze the inter-frame first-order difference and inter-frame second-order difference of the logarithmic Mel spectrum to obtain inter-frame first-order difference features and inter-frame second-order difference features respectively;

[0190] The inter-frame first-order difference feature and the inter-frame second-order difference feature are smoothed to obtain the Mel energy difference coefficient feature.

[0191] In one embodiment, the personalized service generation module 60 is specifically used to:

[0192] generating a safety prompt according to the emotional state, and recording feedback information on the safety prompt;

[0193] Adjusting the content of the safety prompt according to the feedback information;

[0194] The emotional state and the feedback information are stored in the user voiceprint file.

[0195] In one embodiment, the personalized service generation module 60 is specifically used to:

[0196] When it is detected that the user has an abnormal emotional state, candidate service content related to the emotional state is screened out from the personalized service library according to the emotional state;

[0197] Extracting personalized features from the user voiceprint file;

[0198] The target service content that best meets the user's preference is determined from the candidate service contents based on the personalized features.

[0199] In one embodiment, the personalized service generation module 60 is specifically used to:

[0200] Analyze the candidate service content based on the user's historical behavior data and generate a historical behavior score for each candidate service content;

[0201] Analyzing the similarity between the candidate service content and the personalized features, and generating a similarity score for each candidate service content;

[0202] Analyze the satisfaction of the candidate service content according to the historical feedback data of users, and generate a satisfaction score for each candidate service content;

[0203] The historical behavior score, similarity score and satisfaction score are weighted and integrated to obtain a comprehensive score for each candidate service content;

[0204] The candidate service content with the highest comprehensive score is used as the target service content that best meets the user's preference.

[0205] In one embodiment, the emotion fluctuation analysis module 50 is specifically used to:

[0206] Extracting historical emotional state data of the user from the user voiceprint archive, and generating baseline data of a normal emotional state based on the historical emotional state data;

[0207] According to the benchmark data, respectively analyzing the differences between the user's emotional state and a normal emotional state in terms of emotional category, emotional intensity, and emotional duration;

[0208] According to the differences in emotion category, emotion intensity and emotion duration, a mood swing score is generated by combining the weight of each difference;

[0209] According to the emotion fluctuation score and a preset abnormal fluctuation threshold, it is determined whether the user has an abnormal emotion state.

[0210] In one embodiment, the personalized service generation module 60 is specifically used to:

[0211] Collect feedback information and operational behaviors on the target service content;

[0212] Associating the feedback information and operation behavior with the corresponding emotional state and user voiceprint profile to generate a user interaction data set;

[0213] Based on the user interaction dataset, optimizing and updating the parameters of the emotion classification model;

[0214] Based on the user interaction data set, the personalized features of the user voiceprint file are updated.

[0215] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the service side of a service content generation method based on emotion recognition.

[0216] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the user side of a service content generation method based on emotion recognition

[0217] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0218] Collecting voice data and extracting voiceprint features from the voice data;

[0219] Establishing a user voiceprint file according to the voiceprint characteristics;

[0220] Extracting Mel frequency cepstral coefficient features and Mel energy difference coefficient features from the speech data;

[0221] Inputting the Mel frequency cepstral coefficient feature and the Mel energy difference coefficient feature into an emotion classification model, performing emotion classification and recognition through the emotion classification model, and obtaining the user's emotional state;

[0222] Compare the user's emotional state with historical normal emotional states, analyze the user's emotional fluctuations, and determine whether the user has an abnormal emotional state;

[0223] When it is detected that the user has an abnormal emotional state, the target service content is determined from the personalized service library according to the emotional state and the user voiceprint file.

[0224] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0225] Collecting voice data and extracting voiceprint features from the voice data;

[0226] Establishing a user voiceprint file according to the voiceprint characteristics;

[0227] Extracting Mel frequency cepstral coefficient features and Mel energy difference coefficient features from the speech data;

[0228] Inputting the Mel frequency cepstral coefficient feature and the Mel energy difference coefficient feature into an emotion classification model, performing emotion classification and recognition through the emotion classification model, and obtaining the user's emotional state;

[0229] Compare the user's emotional state with historical normal emotional states, analyze the user's emotional fluctuations, and determine whether the user has an abnormal emotional state;

[0230] When it is detected that the user has an abnormal emotional state, the target service content is determined from the personalized service library according to the emotional state and the user voiceprint file.

[0231] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0232] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0233] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0234] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for generating service content based on emotion recognition, characterized in that: The following steps are involved: Collecting voice data and extracting voiceprint features from the voice data; Establishing a user voiceprint file according to the voiceprint characteristics; Extracting Mel frequency cepstral coefficient features and Mel energy difference coefficient features from the speech data; Inputting the Mel frequency cepstral coefficient feature and the Mel energy difference coefficient feature into an emotion classification model, performing emotion classification and recognition through the emotion classification model, and obtaining the user's emotional state; Compare the user's emotional state with historical normal emotional states, analyze the user's emotional fluctuations, and determine whether the user has an abnormal emotional state; When it is detected that the user has an abnormal emotional state, the target service content is determined from the personalized service library according to the emotional state and the user voiceprint file.

2. The method for generating service content based on emotion recognition according to claim 1, characterized in that: Extracting Mel frequency cepstral coefficient features and Mel energy difference coefficient features from the speech data includes: Performing fast Fourier transform on the speech data to obtain a spectrum signal; Performing filtering processing on the spectrum signal by using a Mel filter bank; Perform logarithmic transformation on the filtered spectrum signal to obtain a logarithmic Mel spectrum; Performing discrete cosine transform on the logarithmic Mel frequency spectrum to obtain the Mel frequency cepstral coefficient feature; Analyze the inter-frame first-order difference and inter-frame second-order difference of the logarithmic Mel spectrum to obtain inter-frame first-order difference features and inter-frame second-order difference features respectively; The inter-frame first-order difference feature and the inter-frame second-order difference feature are smoothed to obtain the Mel energy difference coefficient feature.

3. The method for generating service content based on emotion recognition according to claim 1, characterized in that: When the user is detected to have an abnormal emotional state, it also includes: generating a safety prompt according to the emotional state, and recording feedback information on the safety prompt; Adjusting the content of the safety prompt according to the feedback information; The emotional state and the feedback information are stored in the user voiceprint file.

4. The method for generating service content based on emotion recognition according to claim 1, characterized in that: When it is detected that the user has an abnormal emotional state, the target service content is determined from the personalized service library according to the emotional state and the user voiceprint file, including: When it is detected that the user has an abnormal emotional state, candidate service content related to the emotional state is screened out from the personalized service library according to the emotional state; Extracting personalized features from the user voiceprint file; The target service content that best meets the user's preference is determined from the candidate service contents based on the personalized features.

5. The method for generating service content based on emotion recognition according to claim 4, characterized in that: Determining the target service content that best meets the user's preference from the candidate service content based on the personalized features includes: Analyze the candidate service content based on the user's historical behavior data and generate a historical behavior score for each candidate service content; Analyzing the similarity between the candidate service content and the personalized features, and generating a similarity score for each candidate service content; Analyze the satisfaction of the candidate service content according to the historical feedback data of users, and generate a satisfaction score for each candidate service content; The historical behavior score, similarity score and satisfaction score are weighted and integrated to obtain a comprehensive score for each candidate service content; The candidate service content with the highest comprehensive score is used as the target service content that best meets the user's preference.

6. The method for generating service content based on emotion recognition according to claim 1, characterized in that: Comparing the user's emotional state with historical normal emotional states, analyzing the user's emotional fluctuations, and determining whether the user has an abnormal emotional state, including: Extracting historical emotional state data of the user from the user voiceprint archive, and generating baseline data of the normal emotional state based on the historical emotional state data; According to the benchmark data, respectively analyzing the differences between the user's emotional state and a normal emotional state in terms of emotional category, emotional intensity, and emotional duration; According to the differences in emotion category, emotion intensity and emotion duration, a mood swing score is generated by combining the weight of each difference; According to the emotion fluctuation score and a preset abnormal fluctuation threshold, it is determined whether the user has an abnormal emotion state.

7. The method for generating service content based on emotion recognition as claimed in claim 1, characterized in that: When it is detected that the user has an abnormal emotional state, after determining the target service content from the personalized service library according to the emotional state and the user voiceprint file, the method further includes: Collect feedback information and operational behaviors on the target service content; Associating the feedback information and operation behavior with the corresponding emotional state and user voiceprint profile to generate a user interaction data set; Based on the user interaction dataset, optimizing and updating the parameters of the emotion classification model; Based on the user interaction data set, the personalized features of the user voiceprint file are updated.

8. A service content generation device based on emotion recognition, characterized in that: The service content generating device based on emotion recognition comprises: A voiceprint feature extraction module, used to collect voice data and extract voiceprint features from the voice data; A voiceprint file management module, used to establish a user voiceprint file according to the voiceprint characteristics; An emotion feature extraction module, used to extract Mel frequency cepstral coefficient features and Mel energy difference coefficient features from the speech data; An emotion recognition and classification module, used to input the Mel frequency cepstral coefficient feature and the Mel energy difference coefficient feature into an emotion classification model, perform emotion classification and recognition through the emotion classification model, and obtain the user's emotional state; An emotion fluctuation analysis module is used to compare the emotion state of the user with the historical normal emotion state, analyze the emotion fluctuation of the user, and determine whether the user has an abnormal emotion state; The personalized service generation module is used to determine the target service content from the personalized service library according to the emotional state and the user voiceprint file when it is detected that the user has an abnormal emotional state.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a service content generation program based on emotion recognition stored in the memory and executable on the processor. When the service content generation program based on emotion recognition is executed by the processor, the steps of the service content generation method based on emotion recognition as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a service content generation program based on emotion recognition, and when the service content generation program based on emotion recognition is executed by the processor, the steps of the service content generation method based on emotion recognition as described in any one of claims 1 to 7 are implemented.

Images