Intelligent customer service method and device based on multi-modal emotion recognition and storage medium

Abstract

The invention discloses an intelligent customer service method and device based on multi-modal emotion recognition, a storage medium and computer equipment, and the method comprises the steps: starting a camera device and a sound receiving device in response to a video customer service request of a target user, acquiring a facial image of the target user through the camera device and recording voice data of the target user through the sound receiving device; extracting facial features corresponding to the facial image, extracting sound features corresponding to the voice data, and converting the voice data into text information; and in the interaction process of the target user and the intelligent customer service, according to the facial features and the sound features after the intelligent customer service replies the current round of content, dynamically identifying the emotion generated by the target user on the current round of replied content of the intelligent customer service. And determining the next round of reply content of the intelligent customer service based on the emotion recognition result and the next round of text information.

Description

Technical Field

[0001] The present application relates to the field of intelligent customer service technology, and in particular to an intelligent customer service method, device, storage medium and computer equipment based on multimodal emotion recognition. Background Art

[0002] With the rapid development of the Internet and the increasing diversification of user needs, intelligent customer service systems have become an important tool for many companies and platforms to improve service quality and enhance user experience. However, traditional intelligent customer service systems mainly rely on text interaction, which makes it difficult to fully capture and understand the real emotions and intentions of users during communication. This, to a certain extent, limits the intelligence level of intelligent customer service and further improvement of user experience.

[0003] In traditional intelligent customer service systems, when users initiate video customer service requests, the system can only process text messages sent by users, but cannot effectively utilize the rich emotional information contained in videos and audios. This makes it difficult for the system to respond promptly and appropriately to user emotional changes or complex questions, affecting user satisfaction and trust. Summary of the invention

[0004] In view of this, the embodiments of the present application provide an intelligent customer service method, apparatus, storage medium and computer equipment based on multimodal emotion recognition. Through multimodal emotion recognition, the intelligent customer service system can more accurately understand the user's emotional state and needs, thereby providing more intimate and personalized services.

[0005] According to one aspect of the present application, an intelligent customer service method based on multimodal emotion recognition is provided, the method comprising:

[0006] In response to a video customer service request from a target user, a camera device and a sound collection device are activated to obtain a facial image of the target user through the camera device and to record voice data of the target user through the sound collection device;

[0007] Extracting facial features corresponding to the facial image, extracting sound features corresponding to the voice data, and converting the voice data into text information;

[0008] During the interaction between the target user and the intelligent customer service, the emotion of the target user in response to the current round of reply content of the intelligent customer service is dynamically identified based on the facial features and the voice features of the intelligent customer service after the current round of reply content, and the next round of reply content of the intelligent customer service is determined based on the emotion recognition result and the next round of text information.

[0009] In the embodiment of the present application, optionally, the method further includes:

[0010] In response to a video customer service request of a target user, when the target user is wearing a smart monitoring device, obtaining physiological signal data collected by the smart monitoring device, wherein the physiological signal data includes heart rate signal data and blood pressure signal data;

[0011] Extracting physiological characteristics corresponding to the physiological signal data, wherein the physiological characteristics include heart rate characteristics and blood pressure characteristics;

[0012] Accordingly, according to the facial features and the voice features after the intelligent customer service replies to the current round of content, the emotion of the target user generated by the intelligent customer service in response to the current round of content is dynamically identified, including:

[0013] Based on the facial features, voice features and physiological features after the intelligent customer service replies to the current round of content, the emotions of the target user in response to the current round of reply content of the intelligent customer service are dynamically identified, wherein the facial features include position features and morphological features of facial expression feature points, and the voice features include pitch features, speech speed features and volume features.

[0014] In the embodiment of the present application, optionally, dynamically identifying the emotion generated by the target user to the current round of reply content of the intelligent customer service according to the facial features, the voice features and the physiological features after the intelligent customer service replies to the current round of content, includes:

[0015] Using a multimodal feature emotion recognition model, performing multimodal feature fusion on the facial features, the voice features, and the physiological features of the intelligent customer service after the current round of content reply to obtain a fusion feature, and performing emotion recognition on the fusion feature to obtain a first emotion recognition result;

[0016] Using a facial feature emotion recognition model to perform emotion recognition on the facial features of the intelligent customer service after replying to the content of the current round, using a voice feature emotion recognition model to perform emotion recognition on the voice features of the intelligent customer service after replying to the content of the current round, using a physiological feature emotion recognition model to perform emotion recognition on the physiological features of the intelligent customer service after replying to the content of the current round, and fusing the facial feature emotion recognition results, the voice feature emotion recognition results, and the physiological feature emotion recognition results to obtain a second emotion recognition result;

[0017] The emotion recognition result is determined according to the first emotion recognition result and the second emotion recognition result.

[0018] In an embodiment of the present application, optionally, the multimodal feature emotion recognition model includes a multimodal feature fusion network and an emotion recognition network; using the multimodal feature emotion recognition model to perform multimodal feature fusion on the facial features, the voice features, and the physiological features of the intelligent customer service after replying to the content of the current round to obtain a fused feature, and performing emotion recognition on the fused feature to obtain a first emotion recognition result, the method further includes:

[0019] Constructing an initial multimodal feature emotion recognition model, wherein the initial multimodal feature emotion recognition model includes an initial multimodal feature fusion network and an initial emotion recognition network;

[0020] Acquire multiple training samples, wherein each of the training samples includes a multimodal feature sample and a sentiment information label corresponding to the multimodal feature sample;

[0021] Performing a round of training on the initial multimodal feature emotion recognition model using the training samples;

[0022] Dividing the training samples into a plurality of training sample groups;

[0023] Freeze the initial emotion recognition network of the initial multimodal feature emotion recognition model, use any one of the training sample groups to train the network parameters of the initial multimodal feature emotion recognition model, and freeze the initial multimodal feature fusion network of the initial multimodal feature emotion recognition model, use any one of the training sample groups to train the network parameters of the initial emotion recognition network, and continue to use other training sample groups to alternately train the initial multimodal feature fusion network and the initial emotion recognition network until the training conditions are met to obtain the multimodal feature emotion recognition model.

[0024] In the embodiment of the present application, optionally, the method further includes:

[0025] After the video customer service ends, the response effect of each round of the intelligent customer service is evaluated according to the emotion recognition results corresponding to the response content of each round;

[0026] The intelligent customer service is optimized according to the target user's text information in each round, the reply content in each round, and the reply effect of the reply content in each round.

[0027] In an embodiment of the present application, optionally, determining the next round of reply content of the intelligent customer service based on the emotion recognition result and the text information includes:

[0028] Generate intelligent customer service prompt information according to the emotion recognition result and the next round of text information, and input the intelligent customer service prompt information into the intelligent customer service large language model to determine the next round of reply text of the intelligent customer service;

[0029] Determine, according to the emotion recognition result, the next round of expression feature information corresponding to the intelligent customer service, wherein the next round of expression feature information includes voice feature information and expression feature information;

[0030] According to the next round reply text and the next round expression feature information, the next round reply content is output through the intelligent customer service.

[0031] In the embodiment of the present application, optionally, the intelligent customer service is optimized according to the target user's text information in each round, the reply content in each round, and the reply effect of the reply content in each round, including:

[0032] Generate interaction effect information between the target user and the intelligent customer service according to the target user's text information in each round, the reply content in each round, and the reply effect of the reply content in each round;

[0033] The interaction effect information is learned through the intelligent customer service big language model to optimize the intelligent customer service big language model.

[0034] According to another aspect of the present application, an intelligent customer service device based on multimodal emotion recognition is provided, the device comprising:

[0035] An acquisition module, configured to respond to a video customer service request of a target user, start a camera device and a sound collection device, obtain a facial image of the target user through the camera device, and record voice data of the target user through the sound collection device;

[0036] An extraction module, used to extract facial features corresponding to the facial image, extract sound features corresponding to the voice data, and convert the voice data into text information;

[0037] The customer service module is used to dynamically identify the emotion of the target user in response to the current round of reply content of the intelligent customer service according to the facial features and voice features of the intelligent customer service after the current round of reply during the interaction between the target user and the intelligent customer service, and determine the next round of reply content of the intelligent customer service based on the emotion recognition result and the next round of text information.

[0038] In the embodiment of the present application, optionally, the acquisition module is further used to respond to the video customer service request of the target user, and when the target user wears the smart monitoring device, acquire the physiological signal data collected by the smart monitoring device, wherein the physiological signal data includes heart rate signal data and blood pressure signal data;

[0039] The extraction module is further used to extract physiological features corresponding to the physiological signal data, wherein the physiological features include heart rate features and blood pressure features;

[0040] The customer service module is also used to dynamically identify the emotions of the target user in response to the current round of replies of the intelligent customer service based on the facial features, voice features and physiological features after the intelligent customer service replies to the current round of content, wherein the facial features include position features and morphological features of facial expression feature points, and the voice features include pitch features, speech speed features and volume features.

[0041] In the embodiment of the present application, optionally, the customer service module is further used to:

[0042] Using a multimodal feature emotion recognition model, performing multimodal feature fusion on the facial features, the voice features, and the physiological features of the intelligent customer service after the current round of content reply to obtain a fusion feature, and performing emotion recognition on the fusion feature to obtain a first emotion recognition result;

[0043] Using a facial feature emotion recognition model to perform emotion recognition on the facial features of the intelligent customer service after replying to the content of the current round, using a voice feature emotion recognition model to perform emotion recognition on the voice features of the intelligent customer service after replying to the content of the current round, using a physiological feature emotion recognition model to perform emotion recognition on the physiological features of the intelligent customer service after replying to the content of the current round, and fusing the facial feature emotion recognition results, the voice feature emotion recognition results, and the physiological feature emotion recognition results to obtain a second emotion recognition result;

[0044] The emotion recognition result is determined according to the first emotion recognition result and the second emotion recognition result.

[0045] In an embodiment of the present application, optionally, the multimodal feature emotion recognition model includes a multimodal feature fusion network and an emotion recognition network; the customer service module is further used to:

[0046] Constructing an initial multimodal feature emotion recognition model, wherein the initial multimodal feature emotion recognition model includes an initial multimodal feature fusion network and an initial emotion recognition network;

[0047] Acquire multiple training samples, wherein each of the training samples includes a multimodal feature sample and a sentiment information label corresponding to the multimodal feature sample;

[0048] Performing a round of training on the initial multimodal feature emotion recognition model using the training samples;

[0049] Dividing the training samples into a plurality of training sample groups;

[0050] Freeze the initial emotion recognition network of the initial multimodal feature emotion recognition model, use any one of the training sample groups to train the network parameters of the initial multimodal feature emotion recognition model, and freeze the initial multimodal feature fusion network of the initial multimodal feature emotion recognition model, use any one of the training sample groups to train the network parameters of the initial emotion recognition network, and continue to use other training sample groups to alternately train the initial multimodal feature fusion network and the initial emotion recognition network until the training conditions are met to obtain the multimodal feature emotion recognition model.

[0051] In the embodiment of the present application, optionally, the customer service module is further used to:

[0052] After the video customer service ends, the response effect of each round of the intelligent customer service is evaluated according to the emotion recognition results corresponding to the response content of each round;

[0053] The intelligent customer service is optimized according to the target user's text information in each round, the reply content in each round, and the reply effect of the reply content in each round.

[0054] In the embodiment of the present application, optionally, the customer service module is further used to:

[0055] Generate intelligent customer service prompt information according to the emotion recognition result and the next round of text information, and input the intelligent customer service prompt information into the intelligent customer service large language model to determine the next round of reply text of the intelligent customer service;

[0056] Determine, according to the emotion recognition result, the next round of expression feature information corresponding to the intelligent customer service, wherein the next round of expression feature information includes voice feature information and expression feature information;

[0057] According to the next round reply text and the next round expression feature information, the next round reply content is output through the intelligent customer service.

[0058] In the embodiment of the present application, optionally, the customer service module is further used to:

[0059] Generate interaction effect information between the target user and the intelligent customer service according to the target user's text information in each round, the reply content in each round, and the reply effect of the reply content in each round;

[0060] The interaction effect information is learned through the intelligent customer service big language model to optimize the intelligent customer service big language model.

[0061] According to another aspect of the present application, a storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned intelligent customer service method based on multimodal emotion recognition is implemented.

[0062] According to another aspect of the present application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the above-mentioned intelligent customer service method based on multimodal emotion recognition when executing the program.

[0063] Through the above-mentioned technical scheme, the embodiments of the present application provide an intelligent customer service method, device, storage medium and computer equipment based on multimodal emotion recognition. Through multimodal emotion recognition, the intelligent customer service system can more accurately understand the user's emotional state and needs, thereby providing more intimate and personalized services. Moreover, since the system can perceive the user's emotional changes in real time and adjust the reply strategy and content accordingly, it can solve the user's problems or doubts more quickly, reduce ineffective communication and waiting time, and improve interaction efficiency.

[0064] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0066] Figure 1 A schematic diagram of a process of an intelligent customer service method based on multimodal emotion recognition provided in an embodiment of the present application is shown;

[0067] Figure 2 A structural schematic diagram of an intelligent customer service device based on multimodal emotion recognition provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0068] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0069] In this embodiment, an intelligent customer service method based on multimodal emotion recognition is provided. Figure 1 As shown, the method includes:

[0070] Step 101, in response to a video customer service request of a target user, start a camera device and a sound collecting device, obtain a facial image of the target user through the camera device and record the voice data of the target user through the sound collecting device.

[0071] Step 102: extract facial features corresponding to the facial image, extract sound features corresponding to the voice data, and convert the voice data into text information.

[0072] Step 103, during the interaction between the target user and the intelligent customer service, dynamically identify the emotion of the target user in response to the current round of reply content of the intelligent customer service based on the facial features and the voice features after the intelligent customer service responds to the current round of content, and determine the next round of reply content of the intelligent customer service based on the emotion recognition result and the next round of text information.

[0073] The embodiment of the present application proposes an intelligent customer service method based on multimodal emotion recognition, which aims to improve the intelligent customer service system's perception of user emotions and interactive experience by comprehensively utilizing multimodal information such as video and audio. When the target user initiates a video customer service request, the camera device and the sound collection device are started. The camera device is responsible for capturing the user's facial image, while the sound collection device records the user's voice data. This step is the basis for subsequent emotion recognition, ensuring that the system can obtain intuitive visual and auditory information when the user communicates. Next, the collected facial images and voice data are processed. For facial images, facial recognition technology is used to extract facial features, such as expressions, eyes, etc.; for voice data, sound features, such as pitch, speech speed, volume, etc., are extracted, and the voice data is converted into text information for subsequent processing. This step converts the original multimedia data into a form that can be understood and processed by the computer, providing key input for emotion recognition. During the interaction between the target user and the intelligent customer service, the system will perform dynamic emotion recognition based on the reply content of the current round of the intelligent customer service, combined with the facial features and sound features of the user after the reply. This step uses advanced emotion recognition algorithms to analyze the user's emotional response, such as satisfaction, dissatisfaction, doubt, etc. Based on the results of emotion recognition and the next round of text information (which may be the user's new questions or further inquiries), the system can intelligently determine the next round of responses from the intelligent customer service to better fit the user's emotional state and actual needs.

[0074] By applying the technical solution of this embodiment and through multimodal emotion recognition, the intelligent customer service system can more accurately understand the user's emotional state and needs, thereby providing more intimate and personalized services. This emotion-based service experience can significantly improve user satisfaction and loyalty. Since the system can perceive the user's emotional changes in real time and adjust the response strategy and content accordingly, it can solve the user's problems or doubts more quickly, reduce ineffective communication and waiting time, and improve interaction efficiency.

[0075] In the embodiment of the present application, optionally, the method further includes: in response to a video customer service request of a target user, when the target user is wearing a smart monitoring device, obtaining physiological signal data collected by the smart monitoring device, wherein the physiological signal data includes heart rate signal data and blood pressure signal data; extracting physiological features corresponding to the physiological signal data, wherein the physiological features include heart rate features and blood pressure features;

[0076] Correspondingly, in step 103, the target user's emotion towards the current round of reply content of the intelligent customer service is dynamically identified based on the facial features and the voice features after the intelligent customer service replies to the current round of content, including: dynamically identifying the target user's emotion towards the current round of reply content of the intelligent customer service based on the facial features, the voice features and the physiological features after the intelligent customer service replies to the current round of content, wherein the facial features include positional features and morphological features of facial expression feature points, and the voice features include pitch features, speech speed features and volume features.

[0077] In this embodiment, when the target user wears an intelligent monitoring device (such as a smart watch, a health bracelet, etc.), the system will additionally obtain physiological signal data collected by these devices, mainly including heart rate signal data and blood pressure signal data. These data reflect the user's physiological state and are closely related to emotional changes. Corresponding physiological features are extracted from the collected physiological signal data, including heart rate features and blood pressure features. These features quantify the user's physiological response and provide important supplementary information for emotion recognition. Therefore, when performing emotion recognition, not only facial features and sound features are considered, but also physiological features are added. In this way, the system can integrate the positional features and morphological features of facial expression feature points, tone features, speech speed features, volume features, heart rate features, blood pressure features and other information to perform a more comprehensive and accurate dynamic recognition of the target user's emotions. In addition, it should be noted that all user data used in this application are collected with the user's permission. By introducing physiological signal data and its corresponding physiological features, the embodiment of this application can capture the physiological reactions accompanying the user's emotional changes, thereby more accurately identifying the user's emotional state. This multimodal fusion method reduces the error and uncertainty in single modality recognition. As the accuracy of emotion recognition increases, the intelligent customer service system can more accurately understand the user's emotions and needs, thereby providing more intimate and personalized services. This helps to improve user satisfaction and trust, and further enhance the user experience.

[0078] In the embodiment of the present application, optionally, dynamically identifying the emotion generated by the target user to the current round of reply content of the intelligent customer service according to the facial features, the voice features and the physiological features after the intelligent customer service replies to the current round of content, includes:

[0079] Using a multimodal feature emotion recognition model, performing multimodal feature fusion on the facial features, the voice features, and the physiological features of the intelligent customer service after the current round of content reply to obtain a fusion feature, and performing emotion recognition on the fusion feature to obtain a first emotion recognition result;

[0080] Using a facial feature emotion recognition model to perform emotion recognition on the facial features of the intelligent customer service after replying to the content of the current round, using a voice feature emotion recognition model to perform emotion recognition on the voice features of the intelligent customer service after replying to the content of the current round, using a physiological feature emotion recognition model to perform emotion recognition on the physiological features of the intelligent customer service after replying to the content of the current round, and fusing the facial feature emotion recognition results, the voice feature emotion recognition results, and the physiological feature emotion recognition results to obtain a second emotion recognition result;

[0081] The emotion recognition result is determined according to the first emotion recognition result and the second emotion recognition result.

[0082] In this embodiment, two parallel and complementary methods are used to dynamically identify the target user's emotions for the current round of replies from the intelligent customer service, and the final emotion recognition result is determined by fusing the results of the two methods. This method combines the direct recognition of multimodal feature fusion with the strategy of independent recognition and fusion of single-modal features, aiming to improve the accuracy and robustness of emotion recognition. Method 1 is emotion recognition after multimodal feature fusion: First, the facial features, voice features and physiological features of the intelligent customer service after the current round of content replies are fused by using a multimodal feature emotion recognition model. This step integrates feature information from different modalities into a unified feature space through complex machine learning or deep learning algorithms to form fused features. Then, emotion recognition is performed on the fused features to obtain the first emotion recognition result. Since the fused features contain rich information from multiple modalities, this method can capture the user's emotional state more comprehensively. Method 2 is to fuse the single modal features after independent recognition: the facial feature emotion recognition model, the voice feature emotion recognition model and the physiological feature emotion recognition model are used to perform independent emotion recognition on the facial features, voice features and physiological features of the intelligent customer service after the current round of content reply. Each model focuses on the features of its corresponding modality and outputs the corresponding emotion recognition results. Finally, the facial feature emotion recognition results, the voice feature emotion recognition results and the physiological feature emotion recognition results are fused to obtain the second emotion recognition result. This step may involve a weighted or voting mechanism for the recognition results of different modalities to ensure that the contribution of each modality is reasonably considered. For example, the position features of facial expression feature points, the morphological features of facial expression feature points, the pitch features, the speech speed features, the volume features, the heart rate features and the blood pressure features are respectively compared with the features of the previous period corresponding to the respective feature types to determine the feature change amplitude corresponding to each feature type; the facial feature change amplitude is determined according to the position feature change amplitude of the facial expression feature points and the morphological feature change amplitude of the facial expression points; the voice feature change amplitude is determined according to the pitch feature change amplitude, the speech speed feature change amplitude and the volume feature change amplitude; the physiological feature change amplitude is determined according to the heart rate feature change amplitude and the blood pressure feature change amplitude; the facial feature weight, the voice feature weight and the physiological feature weight are determined according to the facial feature change amplitude, the voice feature weight and the physiological feature change amplitude; according to the facial feature weight, the voice feature weight and the physiological feature weight, the facial feature emotion analysis results, the voice feature emotion analysis results and the physiological feature emotion analysis results are fused to obtain the second emotion state information. Finally, according to the first emotion recognition result (from the emotion recognition after multimodal feature fusion) and the second emotion recognition result (from the fusion after single-modal feature independent recognition), the final emotion recognition result is determined through a decision mechanism (such as weighted average, voting, machine learning model, etc.).This method combines the advantages of two recognition paths, taking into account the interaction between multimodal features while retaining the information of independent recognition of each modality. The embodiment of the present application combines the two methods of multimodal feature fusion and fusion after independent recognition of single modal features to more comprehensively capture the user's emotional state, thereby improving the accuracy of emotion recognition.

[0083] In an embodiment of the present application, optionally, the multimodal feature emotion recognition model includes a multimodal feature fusion network and an emotion recognition network; using the multimodal feature emotion recognition model to perform multimodal feature fusion on the facial features, the voice features, and the physiological features of the intelligent customer service after replying to the content of the current round to obtain a fused feature, and performing emotion recognition on the fused feature to obtain a first emotion recognition result, the method further includes:

[0084] Constructing an initial multimodal feature emotion recognition model, wherein the initial multimodal feature emotion recognition model includes an initial multimodal feature fusion network and an initial emotion recognition network;

[0085] Acquire multiple training samples, wherein each of the training samples includes a multimodal feature sample and a sentiment information label corresponding to the multimodal feature sample;

[0086] Performing a round of training on the initial multimodal feature emotion recognition model using the training samples;

[0087] Dividing the training samples into a plurality of training sample groups;

[0088] Freeze the initial emotion recognition network of the initial multimodal feature emotion recognition model, use any one of the training sample groups to train the network parameters of the initial multimodal feature emotion recognition model, and freeze the initial multimodal feature fusion network of the initial multimodal feature emotion recognition model, use any one of the training sample groups to train the network parameters of the initial emotion recognition network, and continue to use other training sample groups to alternately train the initial multimodal feature fusion network and the initial emotion recognition network until the training conditions are met to obtain the multimodal feature emotion recognition model.

[0089] In this embodiment, first, an initial multimodal feature emotion recognition model is constructed, which consists of an initial multimodal feature fusion network and an initial emotion recognition network. The two networks contain random weights or pre-trained weights in the initial state. Multiple training samples are obtained, each of which contains multimodal feature samples (facial features, voice features, and physiological features) and emotion information labels corresponding to these multimodal feature samples. The emotion information label is used to supervise the training process of the model. The initial multimodal feature emotion recognition model is comprehensively trained using all training samples so that the model can preliminarily learn the mapping relationship between multimodal features and emotion information and perform a preliminary learning. The training samples are divided into multiple training sample groups. This is done so that in the subsequent training process, the multimodal feature fusion network and the emotion recognition network can be trained alternately instead of training the entire model at the same time. Alternating training is then performed: first, the parameters of the initial emotion recognition network are frozen, and only the parameters of the initial multimodal feature fusion network are trained using any one training sample group. The purpose of this step is to optimize the fusion network so that it can better fuse features from different modalities. Then, the parameters of the initial multimodal feature fusion network are frozen, and the parameters of the initial emotion recognition network are trained using the same or different training sample groups. The purpose of this step is to optimize the recognition network so that it can more accurately identify emotional information from the fused features. Repeat the above two steps, alternately train the multimodal feature fusion network and the emotion recognition network until the training conditions are met (such as reaching a preset number of iterations, the performance on the validation set is no longer improved, etc.). In the process of alternating training, different training sample groups can be used to ensure that the model can learn diverse data features. After alternating training, a trained multimodal feature emotion recognition model is obtained. The model can effectively fuse features from different modalities and accurately identify the user's emotional state. The embodiment of the present application can make full use of the complementarity between different modal features and improve the accuracy of emotion recognition by alternately training the multimodal feature fusion network and the emotion recognition network. The alternating training process helps to prevent the model from overfitting to a specific type of features or data, thereby enhancing the robustness and generalization ability of the model. In the alternating training process, the weights or fusion strategies of different modal features can be adjusted as needed to adapt to different emotion recognition tasks and data sets. By dividing the training sample groups and training the network parameters alternately, computing resources can be used efficiently and the model training time can be shortened.

[0090] In an embodiment of the present application, optionally, the method also includes: after the video customer service ends, evaluating the response effect of each round of reply content according to the emotion recognition results corresponding to the reply content of each round of the intelligent customer service; optimizing the intelligent customer service according to the text information of each round of the target user, the reply content of each round, and the response effect of the reply content of each round.

[0091] In this embodiment, an evaluation and optimization mechanism for the response effect of intelligent customer service is introduced. This mechanism aims to evaluate the response effect of intelligent customer service by analyzing the emotion recognition results of each round of response content of the intelligent customer service in the process of video customer service, and optimize the intelligent customer service accordingly to improve its service quality and user experience. Specifically, after the video customer service ends, the system collects and organizes the emotion recognition results corresponding to the response content of each round of intelligent customer service. These results reflect the emotional response of the target user to the response content of each round, which is an important basis for evaluating the response effect. Based on the emotion recognition results, the system evaluates the effect of each round of response content. For example, if the emotion recognition result is positive, the evaluation effect is good, and if the emotion recognition result is negative, the evaluation effect is poor. By comparing and analyzing the evaluation effect, the response effect of intelligent customer service and its impact on user emotions can be intuitively understood. In addition to the emotion recognition results, the system also collects and analyzes the text information of each round provided by the target user in the process of video customer service. This information reflects the specific needs and problems of the user and is an important basis for optimizing the response content of the intelligent customer service. After analyzing the user text information and the response effect of the intelligent customer service, the system can optimize the existing problems and deficiencies. For example, if the emotion recognition results of a certain round of reply content show that the user's emotions are relatively negative, and the reply content does not match the user's text information or is not accurate enough, then the system can adjust the reply strategy or optimize the reply template to improve user satisfaction and the positivity of emotional tendencies. The optimization of intelligent customer service is a continuous process. The system needs to continuously collect new user data and emotion recognition results, conduct effect evaluation and optimization adjustments, so as to gradually improve the service quality and user experience of intelligent customer service. The embodiment of the present application can timely discover and correct existing problems and deficiencies through the evaluation and optimization of the effect of intelligent customer service replies, thereby improving the service quality of intelligent customer service. The optimized intelligent customer service can understand user needs more accurately, express reply content more appropriately, and guide user emotions more effectively, thereby enhancing user satisfaction and trust. The optimization process of intelligent customer service is also a process of continuous improvement of its intelligence level. Through continuous learning and improvement, intelligent customer service can respond to various complex scenarios and user needs more intelligently.

[0092] In an embodiment of the present application, optionally, determining the next round of reply content of the intelligent customer service based on the emotion recognition result and the text information includes: generating intelligent customer service prompt information according to the emotion recognition result and the next round of text information, and inputting the intelligent customer service prompt information into the intelligent customer service large language model to determine the next round of reply text of the intelligent customer service; determining the next round of expression feature information corresponding to the intelligent customer service according to the emotion recognition result, wherein the next round of expression feature information includes sound feature information and expression feature information; outputting the next round of reply content through the intelligent customer service according to the next round of reply text and the next round of expression feature information.

[0093] In this embodiment, according to the emotion recognition results of the current round and the expected text information of the next round (which may be preset guiding questions, suggestions or information updates, etc.), the system first generates an intelligent customer service prompt information. This information is intended to guide the intelligent customer service to make a reply in the next round, and take into account the emotional state of the user so as to interact with the user more appropriately. The generated intelligent customer service prompt information is input into the intelligent customer service large language model. Based on its powerful language understanding and generation capabilities, the large language model generates the corresponding next round reply text according to the prompt information to ensure the language accuracy and fluency of the reply text. Furthermore, according to the emotion recognition results of the current round, the system can further determine the expression feature information that the intelligent customer service should adopt when replying in the next round. These feature information include sound feature information and expression feature information, which are used to guide the intelligent customer service on how to better express its reply content in voice and facial expressions. For example, if the emotion recognition results show that the user is in a positive state, the intelligent customer service may use a more cheerful and friendly voice and expression to reply. Finally, according to the generated next round reply text and the determined expression feature information, animation technology is used to output the reply content of the next round through the virtual intelligent customer service according to the expression feature information. This process combines text information with non-text information such as sound and expressions, making the reply content richer and more vivid, which helps to enhance the user's interactive experience. By combining the emotion recognition results and expression feature information, the embodiment of the present application can make the reply of the intelligent customer service closer to the user's emotional state and needs, making the interaction process more natural and smooth. Intelligent customer service can flexibly adjust its reply content and expression method according to the user's emotional response, thereby enhancing user satisfaction and trust. By introducing the comprehensive application of large language models and expression feature information, intelligent customer service shows a higher level of intelligence in understanding and generating reply content, and can better cope with various complex scenarios and user needs.

[0094] In an embodiment of the present application, optionally, the intelligent customer service is optimized according to the target user's text information in each round, the reply content in each round, and the reply effect of the reply content in each round, including: generating interaction effect information between the target user and the intelligent customer service according to the target user's text information in each round, the reply content in each round, and the reply effect of the reply content in each round; and learning the interaction effect information through the intelligent customer service large language model to optimize the intelligent customer service large language model.

[0095] In this embodiment, first, the system generates a comprehensive interactive effect information based on the target user's text information in each round, the intelligent customer service's reply content in each round, and the reply effect of these reply contents (such as user satisfaction, emotional tendency, etc.). This information not only contains the specific content of the conversation, but also reflects the effect of the conversation and user feedback. Then, the system inputs the generated interactive effect information into the intelligent customer service large language model to let the model learn. By analyzing this information, the large language model can identify which reply content is more popular with users and which reply methods may cause user dissatisfaction or misunderstanding, thereby automatically adjusting its reply strategy and language model parameters. Through the above learning process, the intelligent customer service large language model is optimized. The optimized model can understand user needs more accurately, generate reply content that is more in line with user expectations, and show higher adaptability and flexibility during the conversation process. As the interaction between users and intelligent customer service continues, the system will continuously collect new interactive effect information and use it to further optimize the intelligent customer service large language model. Through continuous learning and improvement, the service quality and user experience of intelligent customer service will continue to improve. By learning and analyzing the interaction effect information between the user and the intelligent customer service, the large language model in the embodiment of the present application can more accurately understand the user's needs and generate corresponding reply content, thereby improving the accuracy of the reply.

[0096] By applying the technical solution of this embodiment, through emotion recognition technology, intelligent customer service can more accurately understand the user's emotional state, thereby generating more intimate and personalized reply content. This emotional interaction method allows users to experience more humane services, enhancing user satisfaction and trust; the optimized intelligent customer service large language model makes the reply content more accurate, smooth, and in line with user expectations, further improving the user experience; intelligent customer service can automatically adjust its reply strategy and expression method based on the user's text information, emotion recognition results, and interaction effect information to adapt to the changes in needs of different users and different scenarios. At the same time, through continuous learning and optimization, the reply quality of intelligent customer service has also been significantly improved, reducing the occurrence of misunderstandings and wrong replies.

[0097] Further, as Figure 1The specific implementation of the method, the embodiment of the present application provides an intelligent customer service device based on multimodal emotion recognition, such as Figure 2 As shown, the device comprises:

[0098] An acquisition module, configured to respond to a video customer service request of a target user, start a camera device and a sound collection device, obtain a facial image of the target user through the camera device, and record voice data of the target user through the sound collection device;

[0099] An extraction module, used to extract facial features corresponding to the facial image, extract sound features corresponding to the voice data, and convert the voice data into text information;

[0100] The customer service module is used to dynamically identify the emotion of the target user in response to the current round of reply content of the intelligent customer service according to the facial features and voice features of the intelligent customer service after the current round of reply during the interaction between the target user and the intelligent customer service, and determine the next round of reply content of the intelligent customer service based on the emotion recognition result and the next round of text information.

[0101] In the embodiment of the present application, optionally, the acquisition module is further used to respond to the video customer service request of the target user, and when the target user wears the smart monitoring device, acquire the physiological signal data collected by the smart monitoring device, wherein the physiological signal data includes heart rate signal data and blood pressure signal data;

[0102] The extraction module is further used to extract physiological features corresponding to the physiological signal data, wherein the physiological features include heart rate features and blood pressure features;

[0103] The customer service module is also used to dynamically identify the emotions of the target user in response to the current round of replies of the intelligent customer service based on the facial features, voice features and physiological features after the intelligent customer service replies to the current round of content, wherein the facial features include position features and morphological features of facial expression feature points, and the voice features include pitch features, speech speed features and volume features.

[0104] In the embodiment of the present application, optionally, the customer service module is further used to:

[0105] Using a multimodal feature emotion recognition model, performing multimodal feature fusion on the facial features, the voice features, and the physiological features of the intelligent customer service after the current round of content reply to obtain a fusion feature, and performing emotion recognition on the fusion feature to obtain a first emotion recognition result;

[0106] Using a facial feature emotion recognition model to perform emotion recognition on the facial features of the intelligent customer service after replying to the content of the current round, using a voice feature emotion recognition model to perform emotion recognition on the voice features of the intelligent customer service after replying to the content of the current round, using a physiological feature emotion recognition model to perform emotion recognition on the physiological features of the intelligent customer service after replying to the content of the current round, and fusing the facial feature emotion recognition results, the voice feature emotion recognition results, and the physiological feature emotion recognition results to obtain a second emotion recognition result;

[0107] The emotion recognition result is determined according to the first emotion recognition result and the second emotion recognition result.

[0108] In an embodiment of the present application, optionally, the multimodal feature emotion recognition model includes a multimodal feature fusion network and an emotion recognition network; the customer service module is further used to:

[0109] Constructing an initial multimodal feature emotion recognition model, wherein the initial multimodal feature emotion recognition model includes an initial multimodal feature fusion network and an initial emotion recognition network;

[0110] Acquire multiple training samples, wherein each of the training samples includes a multimodal feature sample and a sentiment information label corresponding to the multimodal feature sample;

[0111] Performing a round of training on the initial multimodal feature emotion recognition model using the training samples;

[0112] Dividing the training samples into a plurality of training sample groups;

[0113] Freeze the initial emotion recognition network of the initial multimodal feature emotion recognition model, use any one of the training sample groups to train the network parameters of the initial multimodal feature emotion recognition model, and freeze the initial multimodal feature fusion network of the initial multimodal feature emotion recognition model, use any one of the training sample groups to train the network parameters of the initial emotion recognition network, and continue to use other training sample groups to alternately train the initial multimodal feature fusion network and the initial emotion recognition network until the training conditions are met to obtain the multimodal feature emotion recognition model.

[0114] In the embodiment of the present application, optionally, the customer service module is further used to:

[0115] After the video customer service ends, the response effect of each round of the intelligent customer service is evaluated according to the emotion recognition results corresponding to the response content of each round;

[0116] The intelligent customer service is optimized according to the target user's text information in each round, the reply content in each round, and the reply effect of the reply content in each round.

[0117] In the embodiment of the present application, optionally, the customer service module is further used to:

[0118] Generate intelligent customer service prompt information according to the emotion recognition result and the next round of text information, and input the intelligent customer service prompt information into the intelligent customer service large language model to determine the next round of reply text of the intelligent customer service;

[0119] Determine, according to the emotion recognition result, the next round of expression feature information corresponding to the intelligent customer service, wherein the next round of expression feature information includes voice feature information and expression feature information;

[0120] According to the next round reply text and the next round expression feature information, the next round reply content is output through the intelligent customer service.

[0121] In the embodiment of the present application, optionally, the customer service module is further used to:

[0122] Generate interaction effect information between the target user and the intelligent customer service according to the target user's text information in each round, the reply content in each round, and the reply effect of the reply content in each round;

[0123] The interaction effect information is learned through the intelligent customer service big language model to optimize the intelligent customer service big language model.

[0124] It should be noted that for other corresponding descriptions of the functional units involved in the intelligent customer service device based on multimodal emotion recognition provided in the embodiment of the present application, please refer to Figure 1 The corresponding description in the method will not be repeated here.

[0125] The embodiment of the present application also provides a computer device, which can be a personal computer, a server, a network device, etc. The computer device includes a bus, a processor, a memory and a communication interface, and can also include an input and output interface and a display device. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store location information. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in each method embodiment are implemented.

[0126] Those skilled in the art will appreciate that the structure of the above-mentioned computer device is only a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components, or combine certain components, or have a different arrangement of components.

[0127] In one embodiment, a computer-readable storage medium is provided. The computer-readable storage medium may be non-volatile or volatile, and stores a computer program thereon. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0128] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0129] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0130] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0131] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0132] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An intelligent customer service method based on multimodal emotion recognition, characterized in that: The method comprises: In response to a video customer service request from a target user, a camera device and a sound collection device are activated to obtain a facial image of the target user through the camera device and to record voice data of the target user through the sound collection device; Extracting facial features corresponding to the facial image, extracting sound features corresponding to the voice data, and converting the voice data into text information; During the interaction between the target user and the intelligent customer service, the emotion of the target user in response to the current round of reply content of the intelligent customer service is dynamically identified based on the facial features and the voice features of the intelligent customer service after the current round of reply content, and the next round of reply content of the intelligent customer service is determined based on the emotion recognition result and the next round of text information.

2. The method according to claim 1, characterized in that The method further comprises: In response to a video customer service request of a target user, when the target user is wearing a smart monitoring device, obtaining physiological signal data collected by the smart monitoring device, wherein the physiological signal data includes heart rate signal data and blood pressure signal data; Extracting physiological characteristics corresponding to the physiological signal data, wherein the physiological characteristics include heart rate characteristics and blood pressure characteristics; Accordingly, according to the facial features and the voice features after the intelligent customer service replies to the current round of content, the emotion of the target user generated by the intelligent customer service in response to the current round of content is dynamically identified, including: Based on the facial features, voice features and physiological features after the intelligent customer service replies to the current round of content, the emotions of the target user in response to the current round of reply content of the intelligent customer service are dynamically identified, wherein the facial features include position features and morphological features of facial expression feature points, and the voice features include pitch features, speech speed features and volume features.

3. The method according to claim 2, characterized in that Dynamically identifying the emotion of the target user to the current round of reply content of the intelligent customer service according to the facial features, the voice features, and the physiological features after the intelligent customer service replies to the current round of content, including: Using a multimodal feature emotion recognition model, performing multimodal feature fusion on the facial features, the voice features, and the physiological features of the intelligent customer service after the current round of content reply to obtain a fusion feature, and performing emotion recognition on the fusion feature to obtain a first emotion recognition result; Using a facial feature emotion recognition model to perform emotion recognition on the facial features of the intelligent customer service after replying to the content of the current round, using a voice feature emotion recognition model to perform emotion recognition on the voice features of the intelligent customer service after replying to the content of the current round, using a physiological feature emotion recognition model to perform emotion recognition on the physiological features of the intelligent customer service after replying to the content of the current round, and fusing the facial feature emotion recognition results, the voice feature emotion recognition results, and the physiological feature emotion recognition results to obtain a second emotion recognition result; The emotion recognition result is determined according to the first emotion recognition result and the second emotion recognition result.

4. The method according to claim 3, characterized in that The multimodal feature emotion recognition model includes a multimodal feature fusion network and an emotion recognition network; using the multimodal feature emotion recognition model to perform multimodal feature fusion on the facial features, the voice features and the physiological features of the intelligent customer service after the current round of content reply to obtain a fusion feature, and performing emotion recognition on the fusion feature to obtain a first emotion recognition result, the method also includes: Constructing an initial multimodal feature emotion recognition model, wherein the initial multimodal feature emotion recognition model includes an initial multimodal feature fusion network and an initial emotion recognition network; Acquire multiple training samples, wherein each of the training samples includes a multimodal feature sample and a sentiment information label corresponding to the multimodal feature sample; Performing a round of training on the initial multimodal feature emotion recognition model using the training samples; Dividing the training samples into a plurality of training sample groups; Freeze the initial emotion recognition network of the initial multimodal feature emotion recognition model, use any one of the training sample groups to train the network parameters of the initial multimodal feature emotion recognition model, and freeze the initial multimodal feature fusion network of the initial multimodal feature emotion recognition model, use any one of the training sample groups to train the network parameters of the initial emotion recognition network, and continue to use other training sample groups to alternately train the initial multimodal feature fusion network and the initial emotion recognition network until the training conditions are met to obtain the multimodal feature emotion recognition model.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: After the video customer service ends, the response effect of each round of the intelligent customer service is evaluated according to the emotion recognition results corresponding to the response content of each round; The intelligent customer service is optimized according to the target user's text information in each round, the reply content in each round, and the reply effect of the reply content in each round.

6. The method according to claim 5, characterized in that Determining the next round of reply content of the intelligent customer service based on the emotion recognition result and the text information includes: Generate intelligent customer service prompt information according to the emotion recognition result and the next round of text information, and input the intelligent customer service prompt information into the intelligent customer service large language model to determine the next round of reply text of the intelligent customer service; Determine, according to the emotion recognition result, the next round of expression feature information corresponding to the intelligent customer service, wherein the next round of expression feature information includes voice feature information and expression feature information; According to the next round reply text and the next round expression feature information, the next round reply content is output through the intelligent customer service.

7. The method according to claim 6, characterized in that According to the target user's text information in each round, the reply content in each round, and the reply effect of the reply content in each round, the intelligent customer service is optimized, including: Generate interaction effect information between the target user and the intelligent customer service according to the target user's text information in each round, the reply content in each round, and the reply effect of the reply content in each round; The interaction effect information is learned through the intelligent customer service big language model to optimize the intelligent customer service big language model.

8. An intelligent customer service device based on multimodal emotion recognition, characterized in that: The device comprises: An acquisition module, configured to respond to a video customer service request of a target user, start a camera device and a sound collection device, obtain a facial image of the target user through the camera device, and record voice data of the target user through the sound collection device; An extraction module, used to extract facial features corresponding to the facial image, extract sound features corresponding to the voice data, and convert the voice data into text information; The customer service module is used to dynamically identify the emotion of the target user in response to the current round of reply content of the intelligent customer service according to the facial features and voice features of the intelligent customer service after the current round of reply during the interaction between the target user and the intelligent customer service, and determine the next round of reply content of the intelligent customer service based on the emotion recognition result and the next round of text information.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

Images