Emotion recognition method and device, service evaluation method and device

By combining facial expressions and audio data, integrating multiple emotional data modalities, and using influence factors and algorithm weighted processing, the problem of inaccurate identification of user emotions in a single modal data is solved, and the accuracy and accuracy of emotion detection are improved.

CN115607154BActive Publication Date: 2024-12-06ANHUI IFLYTEK INTELLIGENT SYST
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211168213.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-09-23
Publication Date
2024-12-06
Estimated Expiration
2042-09-23

AI Technical Summary

Technical Problem

The prior art is difficult to accurately identify the user's emotional category through single-modal dialogue data, resulting in large discrimination errors and inability to accurately reflect the user's emotional state.

Method used

Combining facial expression data and audio data, the user's emotions are determined through the data of the two modalities of facial expression and dialogue audio, and the facial emotion data, audio attribute emotion data and audio text emotion data are integrated, and the influence factor and influence coefficient are used for weighting processing. The sequence minimum optimization algorithm is used to determine the actual sentiment data.

Benefits of technology

It improves the accuracy of user emotions detection, accurately reflects the emotional state of the target user, and reduces the error of emotions detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115607154B_ABST
    Figure CN115607154B_ABST
Patent Text Reader

Abstract

The present application provides an emotion recognition method and device, and a service evaluation method and device, which relate to the field of data processing. The emotion recognition method includes: in response to an emotion recognition request, determining the facial expression data and audio data of the target user corresponding to the emotion recognition request; based on the facial expression data, determining the facial emotion data of the target user; based on the audio data, determining the audio attribute emotion data and audio text emotion data of the target user; based on the facial emotion data, the audio attribute emotion data and the audio text emotion data, determining the actual emotion data of the target user. The embodiment of the present application effectively improves the accuracy of user emotion detection by combining facial emotion data, audio attribute emotion data and audio text emotion data to determine the actual emotion data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and specifically to an emotion recognition method and device, and a service evaluation method and device. Background Art

[0002] Related emotion recognition technologies usually rely on a large amount of conversation data to deeply analyze the main emotion types and distribution of various conversation scenarios. Then, based on big data and deep learning, they actively learn word meanings and word order relationships, and automatically identify the emotion categories expressed in the current conversation and their corresponding confidence levels.

[0003] However, determining the user's emotion category through single-modal conversation data will produce large discrimination errors and thus fail to accurately reflect the user's emotional state. Summary of the invention

[0004] In order to solve the above technical problems, the present application is proposed. The embodiments of the present application provide an emotion recognition method and device, and a service evaluation method and device.

[0005] In a first aspect, an embodiment of the present application provides an emotion recognition method, including: in response to an emotion recognition request, determining the facial expression data and audio data of a target user corresponding to the emotion recognition request; based on the facial expression data, determining the facial emotion data of the target user; based on the audio data, determining the audio attribute emotion data and audio text emotion data of the target user; based on the facial emotion data, the audio attribute emotion data and the audio text emotion data, determining the actual emotion data of the target user.

[0006] In combination with the first aspect, in certain implementations of the first aspect, the actual emotion data of the target user is determined based on facial emotion data, audio attribute emotion data, and audio text emotion data, including: determining at least one emotion influencing factor corresponding to the target user, and an influence coefficient of each of the at least one emotion influencing factors, wherein the influence coefficient is determined based on a parameter value of the emotion influencing factor and an adjustment coefficient corresponding to the parameter value; based on at least one emotion influencing factor and an influence coefficient of each of the at least one emotion influencing factors, respectively processing the facial emotion data, audio attribute emotion data, and audio text emotion data to obtain facial emotion label data corresponding to the facial emotion data, audio attribute emotion label data corresponding to the audio attribute emotion data, and audio text emotion label data corresponding to the audio text emotion data; based on the facial emotion label data, the audio attribute emotion label data, and the audio text emotion label data, determining the actual emotion data of the target user.

[0007] In combination with the first aspect, in certain implementations of the first aspect, based on at least one emotion influencing factor and the influence coefficient of each of at least one emotion influencing factor, facial emotion data, audio attribute emotion data, and audio text emotion data are processed separately to obtain facial emotion label data corresponding to the facial emotion data, audio attribute emotion label data corresponding to the audio attribute emotion data, and audio text emotion label data corresponding to the audio text emotion data, including: based on at least one emotion influencing factor and the influence coefficient of each of at least one emotion influencing factor, respectively determine the influence coefficient fusion data of the facial emotion data, audio attribute emotion data, and audio text emotion data; based on the influence coefficient fusion data of the facial emotion data, audio attribute emotion data, and audio text emotion data, respectively process the facial emotion data, audio attribute emotion data, and audio text emotion data to obtain facial emotion label data, audio attribute emotion label data, and audio text emotion label data.

[0008] In combination with the first aspect, in certain implementations of the first aspect, the actual emotion data of the target user is determined based on facial emotion label data, audio attribute emotion label data and audio text emotion label data, including: based on at least one emotion influencing factor, using the efficacy coefficient method to respectively determine the facial feature weight coefficient corresponding to the facial emotion label data, the audio attribute feature weight coefficient corresponding to the audio attribute emotion label data, and the audio text feature weight coefficient corresponding to the audio text emotion label data; weighting the facial emotion label data based on the facial feature weight coefficient to obtain facial emotion weighted data; weighting the audio attribute emotion label data based on the audio attribute feature weight coefficient to obtain audio attribute emotion weighted data; weighting the audio text emotion label data based on the audio text feature weight coefficient to obtain audio text emotion weighted data; determining the actual emotion data of the target user based on the facial emotion weighted data, the audio attribute emotion weighted data and the audio text emotion weighted data.

[0009] In combination with the first aspect, in certain implementations of the first aspect, the actual emotion data of the target user is determined based on facial emotion weighted data, audio attribute emotion weighted data and audio text emotion weighted data, including: respectively determining the posterior probability of facial emotion weighted data, the posterior probability of audio attribute emotion weighted data, and the posterior probability of audio text emotion weighted data; based on the facial emotion weighted data and the posterior probability of facial emotion weighted data, the audio attribute emotion weighted data and the posterior probability of audio attribute emotion weighted data, the audio text emotion weighted data and the posterior probability of audio text emotion weighted data, using a sequence minimum optimization algorithm to determine the actual emotion data.

[0010] In the second aspect, an embodiment of the present application provides a service evaluation method, comprising: based on the method described in the first aspect, determining the actual emotional data of the business processing user when handling the business; based on the actual emotional data, generating a service evaluation result of the business processing user on the business staff.

[0011] In combination with the second aspect, in certain implementation methods of the second aspect, based on the actual emotion data, a service evaluation result of the business processing user on the business staff is generated, including: obtaining the basic information of the business processing user, the professional information of the staff, and the business processing process information; based on the basic information, professional information and processing process information, determining the theoretical emotion data of the business processing user; based on the theoretical emotion data and the actual emotion data, generating the service evaluation result of the business processing user on the business staff.

[0012] In combination with the second aspect, in certain implementation methods of the second aspect, a service evaluation result of the business processing user on the business staff is generated based on the theoretical emotion data and the actual emotion data, including: if the relative difference between the theoretical emotion data and the actual emotion data is within a preset deviation range, then the service evaluation result is generated based on the actual emotion data; if the relative difference between the theoretical emotion data and the actual emotion data is not within the preset deviation range, then the weighted average emotion data corresponding to the actual emotion data is determined; and the service evaluation result is determined based on the weighted average emotion data.

[0013] In a third aspect, an embodiment of the present application provides an emotion recognition device, comprising: a first determination module, for determining, in response to an emotion recognition request, the facial expression data and audio data of a target user corresponding to the emotion recognition request; a second determination module, for determining the facial emotion data of the target user based on the facial expression data; a third determination module, for determining the audio attribute emotion data and audio text emotion data of the target user based on the audio data; and a fourth determination module, for determining the actual emotion data of the target user based on the facial emotion data, the audio attribute emotion data and the audio text emotion data.

[0014] In the fourth aspect, an embodiment of the present application provides a service evaluation device, including: a determination module, used to determine the actual emotional data of the business processing user when handling the business based on the method described in the first aspect; a generation module, used to generate the service evaluation results of the business processing user on the business staff based on the actual emotional data.

[0015] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program for executing the methods described in the first and second aspects.

[0016] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising: a processor; a memory for storing processor executable instructions; the processor is used to execute the methods described in the first aspect and the second aspect.

[0017] The emotion recognition method provided by the embodiment of the present application may have the following beneficial effects:

[0018] On the one hand, this application determines the facial expression data and audio data of the target user, and jointly determines the user's emotions through the data of the two modalities of facial expression and conversation audio, thereby improving the accuracy of the target user's emotion detection. On the other hand, the audio attribute emotion data and audio text emotion data of the target user are determined through audio data, and then the facial emotion data, audio attribute emotion data and audio text emotion data are fused to determine the actual emotion data of the target user, which once again ensures the accuracy of the target user's emotion detection, thereby accurately reflecting the target user's emotional state. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0020] Figure 1 Shown is a schematic diagram of an implementation environment applicable to an embodiment of the present application.

[0021] Figure 2 Shown is a flow chart of an emotion recognition method provided by an exemplary embodiment of the present application.

[0022] Figure 3 Shown is a schematic diagram of a process for determining actual emotion data of a target user provided by an exemplary embodiment of the present application.

[0023] Figure 4 The figure is a schematic diagram of a process for determining facial emotion label data, audio attribute emotion label data, and audio text emotion label data provided by an exemplary embodiment of the present application.

[0024] Figure 5 Shown is a schematic diagram of a process for determining actual emotion data of a target user provided by another exemplary embodiment of the present application.

[0025] Figure 6 Shown is a schematic diagram of a process for determining actual emotion data of a target user provided by yet another exemplary embodiment of the present application.

[0026] Figure 7Shown is a flow chart of a service evaluation method provided by an exemplary embodiment of the present application.

[0027] Figure 8 Shown is a flowchart of generating service evaluation results provided by an exemplary embodiment of the present application.

[0028] Fig. 9 Shown is a flowchart of generating service evaluation results provided by another exemplary embodiment of the present application.

[0029] Fig.10 Shown is a schematic diagram of the structure of an emotion recognition device provided by an exemplary embodiment of the present application.

[0030] Fig.11 Shown is a schematic diagram of the structure of a service evaluation device provided by an exemplary embodiment of the present application.

[0031] Fig.12 Shown is a schematic structural diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0033] Application Overview

[0034] Related speech recognition technologies usually use rhythmic and spectral acoustic features, as well as some other feature extraction technologies, to judge the user's current emotions through characteristic values ​​such as volume, speech speed, and tone. Among them, spectral acoustic features include formant frequency, speech spectrum energy, speech rate and fundamental frequency, etc. Feature extraction technologies include Mel-Frequency Cepstral Coefficients (MFCC), Linear Predictive Cepstral Coding (LPCC), etc.

[0035] Related facial expression recognition technology uses a camera to capture and locate key parts of the human face (such as the corners of the eyebrows, the tip of the nose, the corners of the mouth, etc.), analyzes the changes in the pixel position of the area to which each key part belongs, and uses a machine learning algorithm to combine the facial expression activities measured in each area to analyze the extracted feature values ​​to reflect people's emotional changes.

[0036] Furthermore, the service evaluation system can accurately understand the feelings and demands of enterprises and the public on the services provided by staff. Based on this, it can accept social supervision and improve service quality in a targeted manner, thus improving people's work efficiency and optimizing the business environment.

[0037] Furthermore, the method of determining the user's emotional data through single-modal audio data or facial expression data is not accurate enough, which may cause errors in the obtained emotional data, and thus fail to accurately predict the user's emotional changes. Based on the above, an embodiment of the present application provides an emotion recognition method. On the one hand, the present application determines the facial expression data and audio data of the target user, and jointly determines the user's emotions through the two-modal data of facial expression and conversation audio, thereby improving the accuracy of user emotion detection. On the other hand, the audio attribute emotion data and audio text emotion data of the target user are determined through audio data, and then the facial emotion data, audio attribute emotion data and audio text emotion data are fused to determine the actual emotion data of the target user, which once again ensures the accuracy of user emotion detection, thereby accurately reflecting the emotional state of the target user.

[0038] Example scenarios

[0039] Figure 1 FIG. 1 is a schematic diagram of an implementation environment applicable to the embodiment of the present application. Figure 1 As shown, the implementation environment includes an image acquisition device 11, an audio acquisition device 12 and a computer device 13, and the image acquisition device 11, the audio acquisition device 12 and the computer device 13 are communicatively connected.

[0040] Specifically, the image acquisition device 11 is used to acquire facial expression data of the target user, and the audio acquisition device 12 is used to acquire audio data of the target user. Exemplarily, the image acquisition device 11 is a camera, and the audio acquisition device 12 is a microphone array.

[0041] The computer device 13 is used to receive the facial expression data and audio data collected by the image acquisition device 11 and the audio acquisition device 12. Exemplarily, a facial expression analysis model, an audio attribute analysis model, a voice quality inspection model, and a mood analysis model are deployed in the computer device 13. Specifically, the facial expression data is input into the facial expression analysis model to determine the facial mood data of the target user; the audio data is input into the audio attribute analysis model to determine the audio attribute mood data of the target user; the audio data is converted into text, and the text is input into the voice quality inspection model to obtain the audio text mood data; the facial mood data, the audio attribute mood data, and the audio text mood data are input into the mood analysis model to obtain the actual mood data of the target user.

[0042] The computer device 13 may be a general-purpose computer or a computer device composed of a dedicated integrated circuit, and this application does not limit this. For example, the computer device 13 may be a mobile terminal device such as a tablet computer, or a personal computer, and the number of computer devices 13 may be one or more, and their types may be the same or different. This embodiment of the application does not limit the number and type of computer devices 13.

[0043] Exemplary Methods

[0044] Figure 2 FIG. 1 is a flow chart of an emotion recognition method provided by an exemplary embodiment of the present application. For example, the emotion recognition method in the embodiment of the present application is applied to a service evaluation system of a service user on the process of handling matters by a staff member. Figure 2 As shown, the emotion recognition method provided in the embodiment of the present application includes the following steps.

[0045] Step S210 , in response to the emotion recognition request, determining facial expression data and audio data of a target user corresponding to the emotion recognition request.

[0046] Specifically, the emotion recognition request can be issued by the target user or by a staff member in the same scene as the target user.

[0047] Exemplarily, the face image of the target user can be collected by a camera, and the face image can be used as the facial expression data of the target user. In addition, the audio data of the target user can be collected by an audio collection device such as a microphone array. Further, based on the face image collected by the camera in real time, the face image is analyzed by an image quality detection algorithm to determine the quality of the face image. If the image quality does not meet the standard, the service evaluation system will issue a prompt so that people can adjust the position and light of the camera according to the prompt to improve the quality of the collected face image. Further, the audio quality detection algorithm is used to analyze the collected audio data to determine the audio quality. And if the audio quality does not meet the standard, the service evaluation system sends relevant prompts to people to remind them to adjust the volume or position of their speech.

[0048] Step S220, determining facial emotion data of the target user based on the facial expression data;

[0049] Exemplarily, the facial expression data is input into the facial expression analysis model to obtain the facial emotion data F of the target user. [m] .

[0050] Step S230: determining the audio attribute emotion data and audio text emotion data of the target user based on the audio data.

[0051] Specifically, audio features are extracted according to the audio data, wherein the audio features include Mel cepstral coefficient features c [m] , pitch feature v [m] , intonation features [m] , speech rate features [m] Based on the Mel cepstral coefficient feature c [m] , pitch feature v [m] , intonation features [m] , speech rate features [m], Using the audio attribute analysis model, determine the audio attribute emotion data V [m] Furthermore, the audio data is converted into audio text in real time, and the audio text is submitted to the speech quality inspection model for emotional keyword similarity comparison to obtain the audio text emotional data T [m] .

[0052] Step S240, determining the actual emotion data of the target user based on the facial emotion data, the audio attribute emotion data and the audio text emotion data.

[0053] Exemplarily, facial emotion data, audio attribute emotion data, and audio text emotion data are submitted to a emotion analysis model, and the actual emotion data of the target user is finally determined.

[0054] Based on the above scheme in the embodiment of the present application, on the one hand, the facial expression data and audio data of the target user are determined, and the user's emotions are jointly determined through the data of the two modalities of facial expression and conversation audio, thereby improving the accuracy of the target user's emotion detection. On the other hand, the audio attribute emotion data and audio text emotion data of the target user are determined through audio data, and then the facial emotion data, audio attribute emotion data and audio text emotion data are fused to determine the actual emotion data of the target user, which once again ensures the accuracy of the target user's emotion detection, thereby accurately reflecting the emotional state of the target user.

[0055] Figure 3 The figure shows a flow chart of determining the actual emotion data of the target user provided by an exemplary embodiment of the present application. Figure 2 Based on the embodiment shown, Figure 3 The embodiment shown is described below in detail. Figure 3 The embodiment shown and Figure 2 The differences and similarities of the illustrated embodiments are not described in detail.

[0056] like Figure 3 As shown, in an embodiment of the present application, determining the actual emotion data of the target user based on facial emotion data, audio attribute emotion data, and audio text emotion data includes the following steps.

[0057] Step S310: determining at least one emotion influencing factor corresponding to the target user and an influence coefficient of each of the at least one emotion influencing factors.

[0058] The influence coefficient is determined based on the parameter value of the emotion influence factor and the adjustment coefficient corresponding to the parameter value. Exemplarily, the influence factor includes a subjective influence factor and an objective influence factor. Exemplarily, the subjective influence factor includes facial contour, the standard Mandarin level of the staff, the current emotional state of the user, etc. The objective influence factor includes queuing time, noise, weather conditions, etc.

[0059] Furthermore, the parameter value of each emotion influencing factor corresponds to a value interval of the emotion influencing factor. For example, the unit of the queuing time is minutes, and the parameter value is 0.01 when the queuing time value interval is (0, 3], the parameter value is 0.02 when the queuing time value interval is (3, 8], the parameter value is 0.04 when the queuing time value interval is (9, 15], and the parameter value is 0.06 when the queuing time value interval is (15, 30]. That is, the longer the queuing time, the greater the impact on people's emotions.

[0060] Furthermore, the adjustment coefficient includes an amplification coefficient and a weight coefficient. The adjustment coefficient is used to calculate the dynamic impact of the parameter value of a certain type of influencing factor on the facial expression data and / or audio attribute emotion data and / or audio text emotion data of the target user in different scenarios. In addition, the amplification coefficient and the weight coefficient can be adjusted at the same time, or the amplification coefficient or the weight coefficient can be adjusted separately according to the specific scenario. Exemplarily, for each influencing factor, its corresponding parameter value is x1, the amplification coefficient corresponding to the parameter value is p1, and the weight coefficient corresponding to the parameter value is f1, then the influence coefficient of the influencing factor is x1p1f1.

[0061] Exemplarily, for the influencing factor of queue time, some target users habitually have stiff or smiling expressions, so when calculating the facial emotion score through facial expression data, the weight coefficient of queue time can be reduced to reduce the influence of queue time on facial emotion score. Exemplarily, the target user complains that there are few service windows and it takes nearly an hour to queue before it is his turn. At this time, the influence of queue time on emotion data can be increased by increasing the amplification coefficient, and the emotion data includes at least one of facial emotion data, audio attribute emotion data, and audio text emotion data. Exemplarily, for the influencing factor of Mandarin level, when calculating facial emotion data, the influence of Mandarin standard degree on facial expression data is very small. At this time, the influence of Mandarin level on facial emotion data can be reduced by reducing the weight coefficient of Mandarin level. When calculating audio text emotion data, it is greatly affected by the standard degree of Mandarin, so it is necessary to increase the amplification coefficient to adjust the parameter value of Mandarin level, thereby increasing the influence of Mandarin level on audio text emotion data.

[0062] Step S320, based on at least one emotion influencing factor and the influence coefficient of each of the at least one emotion influencing factor, respectively process the facial emotion data, the audio attribute emotion data and the audio text emotion data to obtain facial emotion label data corresponding to the facial emotion data, audio attribute emotion label data corresponding to the audio attribute emotion data, and audio text emotion label data corresponding to the audio text emotion data.

[0063] Specifically, facial emotion data, audio attribute emotion data, and audio text emotion data are calibrated through emotion influencing factors and influence coefficients corresponding to the emotion influencing factors to obtain facial emotion label data, audio attribute emotion label data, and audio text emotion label data.

[0064] Step S330, determining the actual emotion data of the target user based on the facial emotion label data, the audio attribute emotion label data and the audio text emotion label data.

[0065] Exemplarily, the trained network model can be used to determine the actual emotion data of the target user based on the facial emotion tag data, the audio attribute emotion tag data, and the audio text emotion tag data. In addition, the actual emotion data of the target user can also be determined using a code algorithm.

[0066] In an embodiment of the present application, by determining emotion influencing factors and influence coefficients of emotion influencing factors, facial emotion data, audio attribute emotion data, and audio text emotion data are calibrated to take into account some influencing factors of the target user in the scenario of handling specific matters, and obtain facial emotion label data, audio attribute emotion label data, and audio text emotion label data that more accurately represent the target user's emotions, thereby further improving the accuracy of emotion detection for the target user.

[0067] Figure 4 The figure shows a flow chart of determining facial emotion tag data, audio attribute emotion tag data, and audio text emotion tag data provided by an exemplary embodiment of the present application. Figure 3 Based on the embodiment shown, Figure 4 The embodiment shown is described below in detail. Figure 4 The embodiment shown and Figure 3 The differences and similarities of the illustrated embodiments are not described in detail.

[0068] like Figure 4As shown, in an embodiment of the present application, based on at least one emotion influencing factor and the influence coefficient of each of at least one emotion influencing factor, facial emotion data, audio attribute emotion data and audio text emotion data are processed respectively to obtain facial emotion label data corresponding to the facial emotion data, audio attribute emotion label data corresponding to the audio attribute emotion data, and audio text emotion label data corresponding to the audio text emotion data, including the following steps.

[0069] Step S410, based on at least one emotion influencing factor and the influence coefficient of each of the at least one emotion influencing factor, respectively determine the influence coefficient fusion data of the facial emotion data, the audio attribute emotion data, and the audio text emotion data.

[0070] Exemplarily, the target user corresponds to m emotion influencing factors. Using the above example, the influence coefficients of the m emotion influencing factors can be expressed as x i p i f i , i∈[1,m]. Furthermore, the fusion data of the influence coefficients of m emotion influencing factors on facial emotion data, audio attribute emotion data and audio text emotion data can be expressed as

[0071] Exemplarily, a pre-trained data fusion model can also be used to determine the influence coefficient fusion data of the emotion influencing factors on facial emotion data, audio attribute emotion data and audio text emotion data based on the parameter value, amplification coefficient and weight coefficient corresponding to each of the m emotion influencing factors.

[0072] Step S420, based on the influence coefficient fusion data of the facial emotion data, the audio attribute emotion data and the audio text emotion data, respectively process the facial emotion data, the audio attribute emotion data and the audio text emotion data to obtain facial emotion label data, audio attribute emotion label data and audio text emotion label data.

[0073] For example, for the facial emotion data F [m] , and its corresponding facial emotion label data can be expressed as:

[0074] For audio attribute emotion data V m , and its corresponding audio attribute emotion label data can be expressed as:

[0075] For audio text sentiment data T m , and its corresponding audio text emotion label data can be expressed as:

[0076] In an embodiment of the present application, the influence coefficient fusion data is finally determined through the influence coefficient fusion data of all emotion influencing factors, and the facial emotion label data, audio attribute emotion label data, and audio text emotion label data are simply, quickly, and accurately determined through the influence coefficient fusion data.

[0077] Figure 5 The figure shows a flow chart of determining the actual emotion data of the target user provided by another exemplary embodiment of the present application. Figure 3 Based on the embodiment shown, Figure 5 The embodiment shown is described below in detail. Figure 5 The embodiment shown and Figure 3 The differences and similarities of the illustrated embodiments are not described in detail.

[0078] like Figure 5 As shown, in an embodiment of the present application, determining the actual emotion data of the target user based on facial emotion label data, audio attribute emotion label data and audio text emotion label data includes the following steps.

[0079] Step S510, based on at least one emotion influencing factor, use the efficacy coefficient method to respectively determine the facial feature weight coefficient corresponding to the facial emotion label data, the audio attribute feature weight coefficient corresponding to the audio attribute emotion label data, and the audio text feature weight coefficient corresponding to the audio text emotion label data.

[0080] Specifically, the efficacy coefficient method refers to the relative value between the actual value of each evaluation index and the allowable range of variation of the index. When conducting comprehensive statistical evaluation, the efficacy coefficient method is first used to convert each index into a dimensionless equivalent, and then the arithmetic mean method is used to calculate each efficacy coefficient as the overall comprehensive evaluation value.

[0081] Table 1 Characterization results of various indicators of the efficacy coefficient method

[0082]

[0083] For example, the values ​​of various indicators in the facial expression analysis, audio attribute analysis, and audio text analysis process are shown in Table 1. As shown in Table 1, the satisfaction value of facial contour is 8, and the unacceptable value is 3; the satisfaction value of Mandarin standard is 4, and the unacceptable value is 1; the satisfaction value of emotion is 6, and the unacceptable value is 9; the satisfaction value of queue length is 3, and the unacceptable value is 15; the satisfaction value of noise is 35, and the unacceptable value is 60.

[0084] Exemplarily, for facial expression analysis, the efficacy coefficient of the facial contour index is: For audio attribute analysis, the efficacy coefficient of the facial contour indicator is: For audio text analysis, the efficacy coefficient of the facial contour indicator is:

[0085] Furthermore, for facial expression analysis, the efficacy coefficient of the Mandarin standard index is: Efficacy coefficient of sentiment indicator: The efficiency coefficient of the queue length indicator is: Efficacy coefficient of noise index: Then the efficacy coefficient of facial expression analysis is: Furthermore, according to the above method, the audio attribute feature weight coefficient and the audio text feature weight coefficient can be calculated respectively.

[0086] Step S520, weighting the facial emotion label data based on the facial feature weight coefficient to obtain facial emotion weighted data.

[0087] Step S530: weighting the audio attribute emotion tag data based on the audio attribute feature weight coefficient to obtain audio attribute emotion weighted data.

[0088] Step S540: weighting the audio text emotion tag data based on the audio text feature weight coefficient to obtain audio text emotion weighted data.

[0089] For example, based on the method in step S510, the facial feature weight coefficient is determined to be a, the audio attribute feature weight coefficient is determined to be b, and the audio text feature weight coefficient is determined to be c, then the facial emotion weighted data is F m *a; audio attribute emotion weighted data is V m *b; audio text emotion weighted data is T m *c.

[0090] Step S550, determining the actual emotion data of the target user based on the facial emotion weighted data, the audio attribute emotion weighted data and the audio text emotion weighted data.

[0091] In an embodiment of the present application, the facial emotion weighted data, audio attribute weighted data, and audio text weighted data are first calculated using the efficacy coefficient method, and then the three weighted data are fused to obtain the actual emotion data of the target user. This takes into account the impact of time factors on emotions, and also takes into account the low accuracy and poor robustness of determining the target user's emotions based on single-modal data, thereby improving the accuracy of emotion detection for the target user.

[0092] Figure 6 The figure shows a flow chart of determining the actual emotion data of the target user provided by another exemplary embodiment of the present application. Figure 5 Based on the embodiment shown, Figure 6 The embodiment shown is described below in detail. Figure 6The embodiment shown and Figure 5 The differences and similarities of the illustrated embodiments are not described in detail.

[0093] like Figure 6 As shown, in an embodiment of the present application, determining the actual emotion data of the target user based on facial emotion weighted data, audio attribute emotion weighted data and audio text emotion weighted data includes the following steps.

[0094] Step S610, respectively determining the posterior probability of the facial emotion weighted data, the posterior probability of the audio attribute emotion weighted data, and the posterior probability of the audio text emotion weighted data.

[0095] Specifically, the posterior probability is the probability obtained by obtaining new additional information through investigation or other means and correcting the prior probability using the Bayesian formula.

[0096] Step S620, based on the facial emotion weighted data and the posterior probability of the facial emotion weighted data, the audio attribute emotion weighted data and the posterior probability of the audio attribute emotion weighted data, the audio text emotion weighted data and the posterior probability of the audio text emotion weighted data, using a sequence minimum optimization algorithm, determine the actual emotion data.

[0097] Exemplarily, the posterior probability of facial emotion weighted data, the posterior probability of audio attribute emotion weighted data, and the posterior probability of audio text emotion weighted data are input into the support vector machine fuser for fusion decision, and the Gaussian radial basis function is selected as the kernel function, and the optimal solution obtained by the sequential minimum optimization algorithm is used as the actual emotion data.

[0098] In an embodiment of the present application, a sequential minimum optimization algorithm is used to decompose the maximum optimization problem into multiple small optimization problems for solution, which can shorten the calculation time and determine the optimal actual emotion data about the target user.

[0099] Figure 7 FIG. 1 is a flow chart of a service evaluation method provided by an exemplary embodiment of the present application. Figure 7 As shown, the service evaluation method provided in the embodiment of the present application includes the following steps.

[0100] Step S710, based on the emotion recognition method described in any embodiment of the present application, determine the actual emotion data of the business processing user when processing the business.

[0101] Step S720, based on the actual emotion data, generates a service evaluation result of the business processing user on the business staff.

[0102] Exemplarily, the service evaluation results include “very satisfied”, “satisfied”, “basically satisfied”, “dissatisfied”, “very dissatisfied” and the like.

[0103] In the embodiments of the present application, on the one hand, by using the emotion recognition method described in any embodiment of the present application to determine the actual emotion data of the business processing user when processing the business, the accuracy of user emotion detection can be improved, and the emotional state of the business processing user can be accurately reflected. On the other hand, through the actual emotion data, the service evaluation results of the business processing user on the business staff are actively generated, effectively avoiding the situation of missed evaluation, wrong evaluation, proxy evaluation, and false evaluation.

[0104] Figure 8 FIG. 1 is a flow chart of generating a service evaluation result provided by an exemplary embodiment of the present application. Figure 7 Based on the embodiment shown, Figure 8 The embodiment shown is described below in detail. Figure 8 The embodiment shown and Figure 7 The differences and similarities of the illustrated embodiments are not described in detail.

[0105] like Figure 8 As shown, in an embodiment of the present application, based on actual emotion data, generating a service evaluation result of a business processing user on a business staff includes the following steps.

[0106] Step S810, obtaining basic information of the business processing user, professional information of the staff, and business processing process information.

[0107] For example, the basic information of the business processing user includes the user's gender (D1), user's age D2, user's education level D3, user's occupation D4, etc. The professionalism information of the staff includes the staff's professionalism D5, the number of errands for the matter D6, etc. The business processing process information includes the complexity of the matter D7, whether the matter has been completed D8, the urgency of the matter to the enterprise or the public D9, the duration of the service D 10 etc.

[0108] Furthermore, the coefficient value range of each indicator is defined. For example, the coefficient of user age D2 is divided according to the interval value, such as [18, 25] coefficient is 0.01, (25, 30] coefficient is 0.03, (30, 40] coefficient is 0.05, (40, 50] coefficient is 0.07, (50, 60] coefficient is 0.08, (60, 100] coefficient is 0.09, etc. The duration of this service D 10(Unit: minutes) The coefficients are divided according to the interval values, such as [1, 5] coefficient is 0.01, (5, 10] coefficient is 0.03, (10, 15] coefficient is 0.04, (15, 20] coefficient is 0.06, (20, 30] coefficient is 0.07, (30, 60] coefficient is 0.09. The coefficients of other indicators are defined according to different rules, and the coefficient definition needs to be optimized and adjusted based on a large amount of data collection and model training.

[0109] Step S820, determining the theoretical emotional data of the business processing user based on the basic information, professional information and processing process information.

[0110] Specifically, based on the coefficients corresponding to each indicator in the basic information, the coefficients corresponding to each indicator in the professional information, and the coefficients corresponding to each indicator in the handling process information, the pre-built government service sentiment prediction model is used to calculate the theoretical sentiment data M of the business handling user when handling this matter.

[0111] Step S830, based on the theoretical emotion data and the actual emotion data, generates a service evaluation result of the business processing user on the business staff.

[0112] In an embodiment of the present application, the user's theoretical emotional data is determined through the basic information of the business processing user, the professional information of the staff, and the business processing process information, and the service evaluation result is further determined based on the theoretical emotional data and the actual emotional data, thereby ensuring that the business processing user's evaluation of the service is more objective.

[0113] Fig. 9 FIG. 1 is a flow chart of generating service evaluation results provided by another exemplary embodiment of the present application. Figure 8 Based on the embodiment shown, Fig. 9 The embodiment shown is described below in detail. Fig. 9 The embodiment shown and Figure 8 The differences and similarities of the illustrated embodiments are not described in detail.

[0114] like Fig. 9 As shown, based on the theoretical emotion data and the actual emotion data, generating the service evaluation results of the business processing users on the business staff includes the following steps.

[0115] Step S910, determining whether the relative difference between the theoretical emotion data and the actual emotion data is within a preset deviation range.

[0116] For example, the theoretical emotion data is M, the actual emotion data is E, and the relative difference between the theoretical emotion data and the actual emotion data is The preset deviation range δ∈[0, 0.05].

[0117] Exemplarily, in actual application, if the judgment result of step S910 is yes, that is, the relative difference between the theoretical emotion data and the actual emotion data is within the preset deviation range, then step S920 is executed; if the judgment result of step S910 is no, that is, the relative difference between the theoretical emotion data and the actual emotion data is not within the preset deviation range, then step S930 and step S940 are executed.

[0118] Specifically, the relative difference between the theoretical emotion data and the actual emotion data is within a preset deviation range, i.e. The relative difference between the theoretical emotion data and the actual emotion data is not within the preset deviation range, that is, or

[0119] Step S920: Generate a service evaluation result based on the actual emotion data.

[0120] Specifically, each evaluation level corresponds to a different value interval of the emotion data. For example, when the emotion data is in [100, 90), the evaluation result is very satisfied; when the emotion data is in [90, 80), the evaluation result is satisfied; when the emotion data is in [80, 70), the evaluation result is basically satisfied; when the emotion data is in [70, 60), the evaluation result is dissatisfied; when the emotion data is in [60, 0], the evaluation result is very dissatisfied.

[0121] Exemplarily, if the actual emotion data is 85, the service evaluation result is satisfactory.

[0122] Step S930: determining weighted average emotion data corresponding to the actual emotion data.

[0123] Specifically, the emotion recognition method described in the embodiment of the present application is used to determine the facial emotion label data and the facial feature weight coefficient, the audio attribute emotion label data and the audio attribute feature weight coefficient, the audio text emotion label data and the audio text feature weight coefficient. Then, the facial emotion label data and the facial feature weight coefficient, the audio attribute emotion label data and the audio attribute feature weight coefficient, the audio text emotion label data and the audio text feature weight coefficient are used for weighted operation to determine the weighted average emotion data corresponding to the actual emotion data.

[0124] Step S940: Determine the service evaluation result based on the weighted average emotion data.

[0125] Likewise, the manner of determining the service evaluation result based on the weighted average emotion data is the same as the example in step S920 .

[0126] In the embodiment of the present application, the actual emotion data and the theoretical emotion data are used for relative difference comparison to measure the degree of deviation between the actual emotion data and the theoretical emotion data, and then the service evaluation result with a higher matching degree is determined according to the degree of deviation, based on the actual emotion data or the weighted average emotion data corresponding to the actual emotion data, so as to improve the accuracy of the service evaluation result. And through the user's emotion data, the service evaluation result is automatically given, which truly reflects the feelings and experience of the business handling user on the service process, avoids the omission, misjudgment and false evaluation of the service evaluation, and can further urge the staff to improve their work service awareness and service attitude.

[0127] Exemplary Devices

[0128] Combination of the above Figures 2 to 9 , describes the method embodiment of the present application in detail, and the following is combined with Fig.10 and Fig.11 , describes the device embodiment of the present application in detail. It should be understood that the description of the method embodiment corresponds to the description of the device embodiment, so the parts not described in detail can refer to the previous method embodiment.

[0129] Fig.10 FIG. 1 is a schematic diagram of the structure of an emotion recognition device provided by an exemplary embodiment of the present application. Fig.10 As shown, the emotion recognition device 100 provided in the embodiment of the present application includes:

[0130] A first determination module 1010 is used to determine facial expression data and audio data of a target user corresponding to the emotion recognition request in response to the emotion recognition request;

[0131] A second determination module 1020, for determining facial emotion data of a target user based on the facial expression data;

[0132] The third determination module 1030 is used to determine the audio attribute emotion data and the audio text emotion data of the target user based on the audio data;

[0133] The fourth determination module 1040 is used to determine the actual emotion data of the target user based on the facial emotion data, the audio attribute emotion data and the audio text emotion data.

[0134] In one embodiment of the present application, the fourth determination module 1040 is also used to determine at least one emotion influencing factor corresponding to the target user, and an influence coefficient of each of the at least one emotion influencing factor, wherein the influence coefficient is determined based on the parameter value of the emotion influencing factor and the adjustment coefficient corresponding to the parameter value; based on the at least one emotion influencing factor and the influence coefficient of each of the at least one emotion influencing factor, respectively process the facial emotion data, the audio attribute emotion data, and the audio text emotion data to obtain facial emotion label data corresponding to the facial emotion data, audio attribute emotion label data corresponding to the audio attribute emotion data, and audio text emotion label data corresponding to the audio text emotion data; based on the facial emotion label data, the audio attribute emotion label data, and the audio text emotion label data, determine the actual emotion data of the target user.

[0135] In one embodiment of the present application, the fourth determination module 1040 is also used to determine the influence coefficient fusion data of the facial emotion data, the audio attribute emotion data, and the audio text emotion data based on at least one emotion influencing factor and the influence coefficient of each of the at least one emotion influencing factor; based on the influence coefficient fusion data of the facial emotion data, the audio attribute emotion data, and the audio text emotion data, respectively process the facial emotion data, the audio attribute emotion data, and the audio text emotion data to obtain facial emotion label data, audio attribute emotion label data, and audio text emotion label data.

[0136] In one embodiment of the present application, the fourth determination module 1040 is also used to determine, based on at least one emotion influencing factor, the facial feature weight coefficient corresponding to the facial emotion label data, the audio attribute feature weight coefficient corresponding to the audio attribute emotion label data, and the audio text feature weight coefficient corresponding to the audio text emotion label data using the efficacy coefficient method; weighting the facial emotion label data based on the facial feature weight coefficient to obtain facial emotion weighted data; weighting the audio attribute emotion label data based on the audio attribute feature weight coefficient to obtain audio attribute emotion weighted data; weighting the audio text emotion label data based on the audio text feature weight coefficient to obtain audio text emotion weighted data; and determining the actual emotion data of the target user based on the facial emotion weighted data, the audio attribute emotion weighted data, and the audio text emotion weighted data.

[0137] In one embodiment of the present application, the fourth determination module 1040 is also used to determine the posterior probability of facial emotion weighted data, the posterior probability of audio attribute emotion weighted data, and the posterior probability of audio text emotion weighted data, respectively; based on the facial emotion weighted data and the posterior probability of facial emotion weighted data, the audio attribute emotion weighted data and the posterior probability of audio attribute emotion weighted data, the audio text emotion weighted data and the posterior probability of audio text emotion weighted data, the actual emotion data is determined using a sequential minimum optimization algorithm.

[0138] Fig.11 FIG. 1 is a schematic diagram of the structure of a service evaluation device provided by an exemplary embodiment of the present application. Fig.11 As shown, the service evaluation device 110 provided in the embodiment of the present application includes:

[0139] The determination module 1110 is used to determine the actual emotion data of the service processing user when processing the service based on the emotion recognition method described in the embodiment of the present application;

[0140] The generation module 1120 is used to generate the service evaluation results of the business processing users on the business staff based on the actual emotion data.

[0141] In one embodiment of the present application, the generating module 1120 is further used to obtain basic information of the business handling user, professional information of the staff, and business handling process information; based on the basic information, professional information and handling process information, determine the theoretical emotional data of the business handling user;

[0142] Based on theoretical emotion data and actual emotion data, generate service evaluation results of business processing users on business staff.

[0143] In one embodiment of the present application, the generation module 1120 is also used to generate a service evaluation result based on the actual emotion data if the relative difference between the theoretical emotion data and the actual emotion data is within a preset deviation range; if the relative difference between the theoretical emotion data and the actual emotion data is not within the preset deviation range, determine the weighted average emotion data corresponding to the actual emotion data; and determine the service evaluation result based on the weighted average emotion data.

[0144] Below, reference Fig.12 To describe an electronic device according to an embodiment of the present application. Fig.12 Shown is a schematic structural diagram of an electronic device provided by an exemplary embodiment of the present application.

[0145] like Fig.12 As shown, the electronic device 120 includes one or more processors 1201 and a memory 1202 .

[0146] The processor 1201 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 120 to perform desired functions.

[0147] The memory 1202 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1201 may run the program instructions to implement the methods of the various embodiments of the present application described above and / or other desired functions. Various contents such as facial expression data, audio data, facial emotion data, audio attribute emotion data, and audio text emotion data may also be stored in the computer-readable storage medium.

[0148] In one example, the electronic device 120 may further include: an input device 1203 and an output device 1204 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0149] The input device 1203 may include, for example, a keyboard, a mouse, etc.

[0150] The output device 1204 can output various information to the outside, including facial expression data, audio data, facial emotion data, audio attribute emotion data, and audio text emotion data, etc. The output device 1204 can include, for example, a display, a speaker, a printer, and a communication network and its connected remote output device, etc.

[0151] Of course, to simplify, Fig.12 Only some of the components related to the present application in the electronic device 120 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 120 may also include any other appropriate components.

[0152] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present application described above in this specification.

[0153] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0154] In addition, an embodiment of the present application may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps of the method according to various embodiments of the present application described above in this specification.

[0155] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0156] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0157] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0158] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0159] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0160] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. An emotion recognition method, characterized in that: include: In response to an emotion recognition request, determining facial expression data and audio data of a target user corresponding to the emotion recognition request; Determining facial emotion data of the target user based on the facial expression data; Based on the audio data, determining audio attribute emotion data and audio text emotion data of the target user; Determining actual emotion data of the target user based on the facial emotion data, the audio attribute emotion data, and the audio text emotion data; The determining the actual emotion data of the target user based on the facial emotion data, the audio attribute emotion data and the audio text emotion data comprises: Determining at least one emotion influencing factor corresponding to the target user and an influence coefficient of each of the at least one emotion influencing factor, wherein the influence coefficient is determined based on a parameter value of the emotion influencing factor and an adjustment coefficient corresponding to the parameter value; Based on the at least one emotion influencing factor and the influence coefficient of each of the at least one emotion influencing factor, the facial emotion data, the audio attribute emotion data and the audio text emotion data are processed respectively to obtain facial emotion label data corresponding to the facial emotion data, audio attribute emotion label data corresponding to the audio attribute emotion data, and audio text emotion label data corresponding to the audio text emotion data; Based on the facial emotion label data, the audio attribute emotion label data and the audio text emotion label data, actual emotion data of the target user is determined.

2. The method according to claim 1, characterized in that The method of processing the facial emotion data, the audio attribute emotion data, and the audio text emotion data based on the at least one emotion influencing factor and the influence coefficient of each of the at least one emotion influencing factor to obtain facial emotion label data corresponding to the facial emotion data, audio attribute emotion label data corresponding to the audio attribute emotion data, and audio text emotion label data corresponding to the audio text emotion data includes: Based on the at least one emotion influencing factor and the influence coefficient of each of the at least one emotion influencing factor, respectively determine the influence coefficient fusion data of the facial emotion data, the audio attribute emotion data, and the audio text emotion data; Based on the influence coefficient fusion data of the facial emotion data, the audio attribute emotion data and the audio text emotion data, the facial emotion data, the audio attribute emotion data and the audio text emotion data are processed respectively to obtain the facial emotion label data, the audio attribute emotion label data and the audio text emotion label data.

3. The method according to claim 1, characterized in that The determining the actual emotion data of the target user based on the facial emotion label data, the audio attribute emotion label data and the audio text emotion label data comprises: Based on the at least one emotion influencing factor, using the efficacy coefficient method to respectively determine the facial feature weight coefficient corresponding to the facial emotion label data, the audio attribute feature weight coefficient corresponding to the audio attribute emotion label data, and the audio text feature weight coefficient corresponding to the audio text emotion label data; Performing weighted processing on the facial emotion label data based on the facial feature weight coefficient to obtain facial emotion weighted data; Performing weighted processing on the audio attribute emotion tag data based on the audio attribute feature weight coefficient to obtain audio attribute emotion weighted data; Performing weighted processing on the audio text emotion tag data based on the audio text feature weight coefficient to obtain audio text emotion weighted data; Based on the facial emotion weighted data, the audio attribute emotion weighted data and the audio text emotion weighted data, actual emotion data of the target user is determined.

4. The method according to claim 3, characterized in that The determining the actual emotion data of the target user based on the facial emotion weighted data, the audio attribute emotion weighted data and the audio text emotion weighted data comprises: Respectively determining the posterior probability of the facial emotion weighted data, the posterior probability of the audio attribute emotion weighted data, and the posterior probability of the audio text emotion weighted data; Based on the facial emotion weighted data and the posterior probability of the facial emotion weighted data, the audio attribute emotion weighted data and the posterior probability of the audio attribute emotion weighted data, the audio text emotion weighted data and the posterior probability of the audio text emotion weighted data, the actual emotion data is determined using a sequential minimum optimization algorithm.

5. A service evaluation method, characterized in that: include: Based on the method described in any one of claims 1 to 4, determining the actual emotional data of the business processing user when processing the business; Based on the actual emotion data, a service evaluation result of the business processing user on the business staff is generated.

6. The method according to claim 5, characterized in that Generating the service evaluation result of the business processing user on the business staff based on the actual emotion data includes: Obtaining basic information of the business processing user, professional information of the staff, and information on the processing process of the business; Determining theoretical emotional data of the business handling user based on the basic information, the professional information and the handling process information; Based on the theoretical emotion data and the actual emotion data, a service evaluation result of the business processing user on the business staff is generated.

7. The method according to claim 6, characterized in that The generating, based on the theoretical emotion data and the actual emotion data, a service evaluation result of the service processing user on the service staff, comprises: If the relative difference between the theoretical emotion data and the actual emotion data is within a preset deviation range, generating the service evaluation result based on the actual emotion data; If the relative difference between the theoretical emotion data and the actual emotion data is not within the preset deviation range, determining the weighted average emotion data corresponding to the actual emotion data; Based on the weighted average emotion data, the service evaluation result is determined.

8. An emotion recognition device, characterized in that: include: A first determination module, configured to determine, in response to an emotion recognition request, facial expression data and audio data of a target user corresponding to the emotion recognition request; A second determination module, configured to determine facial emotion data of the target user based on the facial expression data; A third determination module, configured to determine the audio attribute emotion data and audio text emotion data of the target user based on the audio data; A fourth determination module, configured to determine actual emotion data of the target user based on the facial emotion data, the audio attribute emotion data, and the audio text emotion data; The fourth determination module is also used to determine at least one emotion influencing factor corresponding to the target user and an influence coefficient of each of the at least one emotion influencing factors, wherein the influence coefficient is determined based on a parameter value of the emotion influencing factor and an adjustment coefficient corresponding to the parameter value; based on the at least one emotion influencing factor and the influence coefficient of each of the at least one emotion influencing factors, the facial emotion data, the audio attribute emotion data and the audio text emotion data are processed respectively to obtain facial emotion label data corresponding to the facial emotion data, audio attribute emotion label data corresponding to the audio attribute emotion data, and audio text emotion label data corresponding to the audio text emotion data; based on the facial emotion label data, the audio attribute emotion label data and the audio text emotion label data, the actual emotion data of the target user is determined.

9. A service evaluation device, characterized in that: include: A determination module, used to determine the actual emotional data of the business processing user when processing the business based on the method described in any one of claims 1 to 4; A generation module is used to generate a service evaluation result of the business processing user on the business staff based on the actual emotion data.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the method according to any one of claims 1 to 7.

11. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is used to execute the method according to any one of claims 1 to 7.