Customer service seat switching method, device and electronic equipment

Abstract

The present application provides a method, device and electronic device for switching customer service seats. The method includes: determining the emotion, fatigue and professionalism of the artificial agent based on video data, first audio data and first text, and determining the user's satisfaction based on the second text; determining whether the switching conditions are met based on the emotion of the artificial agent, the fatigue of the artificial agent, the professionalism of the artificial agent and the user's satisfaction, and when it is determined that the switching conditions are met, controlling the artificial agent to interrupt the call with the user, and controlling the virtual agent to continue the call with the user. In this solution, based on the combination of multimodal emotion detection, fatigue detection, professionalism detection and text analysis technology, the interaction status between the artificial agent and the user can be analyzed from multiple dimensions, and the scene dialogue of the virtual agent can be seamlessly connected according to the analysis results, thereby realizing accurate automatic switching between virtual agents and artificial agents.

Description

Technical Field

[0001] The present application relates to the technical field of multimodal information processing, and in particular to a method, device, computer-readable storage medium and electronic device for switching customer service seats. Background Art

[0002] There are currently two types of switching between human agents and virtual agents. One is based on the user's own choice and the system's analysis of user demands, judging that virtual agents can provide services, thereby triggering the service mechanism of virtual agents; the second is to actively switch to human agents when it is detected that the user is dissatisfied with the virtual agent service. Both of these situations consider human agents as backup and supplement for virtual agents. However, in the current solution, only virtual agents can be switched to human agents, and it is not possible to automatically switch from human agents to virtual agents. Summary of the invention

[0003] The main purpose of the present application is to provide a method, device, computer-readable storage medium and electronic device for switching customer service seats to solve the problem in the prior art that it is impossible to automatically switch from a manual seat to a virtual seat.

[0004] According to one aspect of an embodiment of the present invention, a method for switching a customer service agent is provided, comprising: obtaining video data and first audio data of an artificial agent during a speech process, and converting the first audio data into a first text; obtaining second audio data of a user during a speech process, and converting the second audio data into a second text, wherein the user requests the artificial agent to provide artificial customer service services, and the artificial agent and the user have a conversation; determining the emotion, fatigue and professionalism of the artificial agent according to the video data, the first audio data and the first text, and determining the user satisfaction according to the second text; determining whether a switching condition is met according to the emotion of the artificial agent, the fatigue of the artificial agent, the professionalism of the artificial agent and the user satisfaction, and if it is determined that the switching condition is met, controlling the artificial agent to interrupt the conversation with the user, and controlling the virtual agent to continue the conversation with the user, wherein the virtual agent refers to a virtual character generated by artificial intelligence, which provides virtual customer service services to the user through a chat robot function.

[0005] Optionally, determining the emotion, fatigue and professionalism of the artificial agent based on the video data, the first audio data and the first text includes: performing feature extraction on the video data to obtain an image feature vector; performing feature extraction on the first audio data to obtain an audio feature vector; performing feature extraction on the first text to obtain a text feature vector; determining the emotion of the artificial agent based on the image feature vector, the audio feature vector and the text feature vector; determining the fatigue of the artificial agent based on the image feature vector and the audio feature vector; and determining the professionalism of the artificial agent based on the first text.

[0006] Optionally, determining the emotion of the artificial agent according to the image feature vector, the audio feature vector and the text feature vector includes: performing feature fusion according to the image feature vector, the audio feature vector and the text feature vector to obtain a first fused feature vector; comparing the first fused feature vector with a plurality of multidimensional emotion feature vectors to determine a first target feature vector having the smallest similarity to the first fused feature vector, and determining the emotion corresponding to the first target feature vector as the target emotion; determining the fatigue of the artificial agent according to the image feature vector and the audio feature vector includes: performing feature fusion according to the image feature vector and the audio feature vector to obtain a second fused feature vector; comparing the second fused feature vector with a plurality of multidimensional fatigue feature vectors to determine a second target feature vector having the smallest similarity to the second fused feature vector, and determining the fatigue corresponding to the second target feature vector as the target fatigue.

[0007] Optionally, determining the professionalism of the human agent based on the first text includes: extracting professional terms and non-professional terms in the first text; and determining the professionalism of the human agent based on a first proportion of the professional terms in the first text and a second proportion of the non-professional terms in the first text.

[0008] Optionally, determining the user's satisfaction based on the second text includes: extracting target keywords in the second text, the target keywords including at least one of the following: words representing anger, words representing dissatisfaction with the service, and words representing the need to change a manual seat; determining the user's satisfaction based on the number of the target keywords in the second text.

[0009] Optionally, determining whether a switching condition is met according to the emotion of the artificial agent, the fatigue of the artificial agent, the professionalism of the artificial agent and the satisfaction of the user includes: obtaining a first score and a first weight coefficient corresponding to the emotion of the artificial agent, the first score is determined according to the emotion of the artificial agent, and the first weight coefficient refers to the importance of the first score in the target score; obtaining a second score and a second weight coefficient corresponding to the fatigue of the artificial agent, the second score is determined according to the fatigue of the artificial agent, and the second weight coefficient refers to the importance of the second score in the target score; obtaining The third score and third weight coefficient corresponding to the professionalism of the artificial agent, the third score is determined according to the professionalism of the artificial agent, and the third weight coefficient refers to the importance of the third score in the target score; the fourth score and fourth weight coefficient corresponding to the user's satisfaction are obtained, the fourth score is determined according to the user's satisfaction, and the fourth weight coefficient refers to the importance of the fourth score in the target score; the target score is determined according to the first score, the first weight coefficient, the second score, the second weight coefficient, the third score, the third weight coefficient, the fourth score, and the fourth weight coefficient. If the target score is greater than the score threshold, it is determined that the switching condition is met, and if the target score is less than or equal to the score threshold, it is determined that the switching condition is not met.

[0010] Optionally, when it is determined that the switching condition is met, the manual agent is controlled to interrupt the call with the user, and the virtual agent is controlled to continue the call with the user, including: obtaining the timbre characteristics and voiceprint characteristics of the manual agent; controlling the manual agent to interrupt the call with the user, and controlling the virtual agent to continue the call with the user based on the timbre characteristics and the voiceprint characteristics, wherein, during the call between the manual agent and the user, the virtual agent monitors the call in real time.

[0011] According to another aspect of an embodiment of the present invention, a switching device for a customer service agent is provided, comprising: a first processing unit, configured to obtain video data and first audio data of an artificial agent during a speaking process, and convert the first audio data into a first text; a second processing unit, configured to obtain second audio data of a user during a speaking process, and convert the second audio data into a second text, wherein the user requests the artificial agent to provide artificial customer service services, and the artificial agent and the user have a conversation; a determination unit, configured to determine the emotion, fatigue and professionalism of the artificial agent according to the video data, the first audio data and the first text, and determine the user satisfaction according to the second text; a third processing unit, configured to determine whether a switching condition is met according to the emotion of the artificial agent, the fatigue of the artificial agent, the professionalism of the artificial agent and the user satisfaction, and if it is determined that the switching condition is met, control the artificial agent to interrupt the conversation with the user, and control the virtual agent to continue the conversation with the user, wherein the virtual agent refers to a virtual character generated by artificial intelligence, which provides virtual customer service services to the user through a chat robot function.

[0012] According to another aspect of the embodiments of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein the program executes any one of the methods described.

[0013] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising: one or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of the methods described.

[0014] In an embodiment of the present invention, firstly, the video data and the first audio data of the artificial agent during the speaking process are obtained, and the first audio data is converted into a first text, then the second audio data of the user during the speaking process is obtained, and the second audio data is converted into a second text, then the emotion, fatigue and professionalism of the artificial agent are determined according to the video data, the first audio data and the first text, and the user's satisfaction is determined according to the second text, and finally, whether the switching conditions are met according to the emotion of the artificial agent, the fatigue of the artificial agent, the professionalism of the artificial agent and the user's satisfaction, and when it is determined that the switching conditions are met, the artificial agent is controlled to interrupt the call with the user, and the virtual agent is controlled to continue the call with the user. In this scheme, based on the combination of multimodal emotion detection, fatigue detection, professionalism detection and text analysis technology, the interaction status between the artificial agent and the user can be analyzed from multiple dimensions, and the scene dialogue of the virtual agent can be seamlessly connected according to the analysis results, so that accurate automatic switching between virtual agents and artificial agents can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0016] Figure 1 A schematic diagram showing a flow chart of a method for switching a customer service seat according to an embodiment of the present application is shown;

[0017] Figure 2 A schematic structural diagram of a switching device for a customer service seat according to an embodiment of the present application is shown;

[0018] Figure 3 A schematic diagram of the structure of a customer service agent switching system is shown;

[0019] Figure 4 A schematic diagram showing the processing flow of the customer service agent switching system. DETAILED DESCRIPTION

[0020] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0021] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] It should be understood that when an element (such as a layer, film, region, or substrate) is described as being "on" another element, the element may be directly on the other element, or there may be intermediate elements. Moreover, in the specification and claims, when an element is described as being "connected" to another element, the element may be "directly connected" to the other element, or "connected" to the other element through a third element.

[0024] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0025] Multimodal emotion detection: The task of detecting emotion categories is achieved by extracting, fusing and identifying features of the video’s images, audio and text respectively.

[0026] Multimodal fatigue detection: The fatigue detection task is achieved by extracting, fusing and recognizing features of the video image and audio respectively.

[0027] At present, multimodal emotion detection has a certain foundation, and has multimodal emotion detection methods including video, audio, text, etc. In the user service process based on intelligent voice customer service, there are also multi-dimensional emotion detection solutions for users. Based on the detection of user audio and video data, multi-dimensional user emotion judgment results are obtained, and then according to the user's emotional state, the human-machine dialogue between the user and the virtual agent is switched to a human-to-human dialogue between the user and the artificial agent.

[0028] At the same time, for fatigue detection, there are currently fatigue detection methods based on video images and semantic fatigue detection methods. However, most of them are applied to the field of fatigue driving detection, and multi-dimensional fatigue detection has not been applied to artificial seat status detection.

[0029] There are currently two types of switching between human agents and virtual agents. One is based on the user's own choice and the system's analysis of user demands, judging that virtual agents can provide services, thereby triggering the service mechanism of virtual agents; the second is to actively switch to human agents when it is detected that the user is dissatisfied with the virtual agent service. Both of these situations consider human agents as backups and supplements for virtual agents. However, it does not cover the automated processing and remedies when users are dissatisfied with human agents or when human agents themselves violate regulations or have abnormal emotions. Not only human agents can serve as backups for virtual agents, but virtual agents can also serve as backups for human agents in certain situations.

[0030] At present, the triggering method for switching between human and virtual seats is relatively simple, mainly when the mood is abnormal or based on the conclusion of user demand analysis, the virtual seat is triggered to switch to a human agent. When there is a situation where a human agent needs to be switched to a virtual agent, there is a lack of a multi-dimensional judgment mechanism for human agents, and a more comprehensive judgment basis is needed to ensure the accuracy of the judgment of switching between virtual and human agents.

[0031] At present, in the process of interaction between human agents and users, there are often ineffective communications and complaints due to the lack of professionalism of human agents or poor communication between human agents and users. At present, most systems with human agent status analysis only give real-time reminders for human agents' illegal emotions and illegal speech, or perform emotion detection and voice analysis on human agent call recordings and then give optimization suggestions. It cannot guarantee that human agents can autonomously adjust their emotions and call status in real time as required, resulting in increasingly fierce confrontation between human agents and users.

[0032] As mentioned in the background technology, it is impossible to automatically switch from a manual seat to a virtual seat in the prior art. In order to solve the above problem, in a typical embodiment of the present application, a method, device, computer-readable storage medium and electronic device for switching a customer service seat are provided.

[0033] According to an embodiment of the present application, a method for switching customer service seats is provided.

[0034] Figure 1 1 is a flow chart of a method for switching customer service seats according to an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0035] Step S101, obtaining video data and first audio data of a human agent during a speaking process, and converting the first audio data into a first text;

[0036] Specifically, a video acquisition device may be used to acquire video data of the artificial agent during the speaking process, and the video acquisition device may be a camera. An audio acquisition device may be used to acquire first audio data of the artificial agent during the speaking process, and the audio acquisition device may be a microphone. The video data may include data such as the facial expression of the artificial agent and the facial blood oxygen protection degree of the artificial agent.

[0037] In the above step S101, information of multiple dimensions can be obtained, specifically, video data, first audio data and first text generated based on the first audio data can be obtained, so that the status of the artificial agent can be evaluated from multiple dimensions later to further ensure that the accuracy of the status detection of the artificial agent is high.

[0038] Step S102, obtaining second audio data of the user during the speaking process, and converting the second audio data into a second text, wherein the user requests the artificial agent to provide artificial customer service services, and the artificial agent and the user have a conversation;

[0039] Specifically, an audio collection device may be used to collect the second audio data of the user during the speaking process.

[0040] In the above step S102, user data can be obtained from the user direction, so that the user satisfaction can be evaluated from the user dimension later, and then it can be further determined whether it is necessary to switch to a virtual seat to interact with the user according to the user satisfaction.

[0041] Step S103, determining the emotion, fatigue and professionalism of the artificial agent according to the video data, the first audio data and the first text, and determining the user's satisfaction according to the second text;

[0042] In the above step S103, based on the multimodal data of the video data, the first audio data and the first text, the change trend of the artificial agent service can be determined, the call status can be predicted, and when the call status shifts from good to bad, a warning can be issued in time, and when the switching conditions are met, the virtual agent can take over the conversation in time to interact with the user. That is, the data between the artificial agent and the user can be collected, and the call status can be evaluated from multiple dimensions of both the artificial agent and the user. In this way, the evaluation dimensions are richer and more complete, which can ensure more accurate subsequent switching.

[0043] Step S104, determining whether the switching condition is met according to the emotion of the artificial agent, the fatigue of the artificial agent, the professionalism of the artificial agent and the satisfaction of the user; if it is determined that the switching condition is met, controlling the artificial agent to interrupt the call with the user, and controlling the virtual agent to continue the call with the user, wherein the virtual agent refers to a virtual character generated by artificial intelligence, which provides virtual customer service services to the user through a chatbot function.

[0044] In the above step S104, through the results of the multi-dimensional analysis of the call status, the manual agent can be switched to the virtual agent in time, and the virtual agent can be controlled to use the synthetic voice of the manual agent to communicate with the user in a good emotional and language state, so that the user can be less aware of the human-machine switch as much as possible, thereby ensuring a good user experience. By using the virtual agent as a backup for the manual agent, the key mitigation of user service quality assurance can be completed, thereby further improving the call efficiency between the customer service agent and the user, and ensuring that the manual agent is always in the best state to communicate with the user, which can reduce the complaint rate of the manual agent.

[0045] In the above method, firstly, the video data and the first audio data of the artificial agent during the speaking process are obtained, and the first audio data is converted into a first text, then the second audio data of the user during the speaking process is obtained, and the second audio data is converted into a second text, then the emotion, fatigue and professionalism of the artificial agent are determined according to the video data, the first audio data and the first text, and the user's satisfaction is determined according to the second text, and finally, whether the switching conditions are met according to the emotion of the artificial agent, the fatigue of the artificial agent, the professionalism of the artificial agent and the user's satisfaction, and when it is determined that the switching conditions are met, the artificial agent is controlled to interrupt the call with the user, and the virtual agent is controlled to continue the call with the user. In this solution, based on the combination of multimodal emotion detection, fatigue detection, professionalism detection and text analysis technology, the interaction status between the artificial agent and the user can be analyzed from multiple dimensions, and the scene dialogue of the virtual agent can be seamlessly connected according to the analysis results, so as to realize accurate automatic switching between virtual agents and artificial agents.

[0046] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0047] Specifically, this solution is based on multi-dimensional real-time analysis of the emotions, service professionalism, fatigue of the artificial seat, and user satisfaction. By performing multi-modal emotion detection, voice analysis and fatigue detection on the video data, first audio data and first text transcribed from the artificial seat, and real-time analysis of the second audio data of the user, when it is judged that the artificial seat is in a bad mood or even has the possibility of offending the user, or the artificial seat lacks professionalism (such as too much saliva, too much chatter, etc.), or the fatigue of the artificial seat exceeds the standard, or the user frequently expresses dissatisfaction, through a comprehensive dimensional evaluation, when the evaluation reaches the switching standard (that is, meets the switching conditions), the virtual seat will take over the conversation in time, use the synthetic voice of the virtual agent to have a human-computer dialogue with the user, and promptly remind the artificial seat to adjust the status. After the artificial seat is adjusted, the human-to-human dialogue between the artificial agent and the user will be reopened, and a good communication experience will be provided to the user through seamless human-computer switching, which can effectively curb the occurrence of user complaints and improve the user experience.

[0048] In order to ensure the accuracy of the detection of the emotion, fatigue and professionalism of the artificial seat, the emotion, fatigue and professionalism of the artificial seat can be detected by using a feature extraction method and a feature vector. In an embodiment of the present application, the emotion, fatigue and professionalism of the artificial seat are determined according to the video data, the first audio data and the first text, specifically including the following steps:

[0049] Step S201, extracting features from the video data to obtain an image feature vector;

[0050] Step S202, extracting features from the first audio data to obtain an audio feature vector;

[0051] Step S203, extracting features from the first text to obtain a text feature vector;

[0052] Step S204, determining the emotion of the artificial agent according to the image feature vector, the audio feature vector and the text feature vector;

[0053] Changes in call status often do not occur instantaneously. Generally, there is a process for the call status to change from good to bad. During this process, the facial expressions corresponding to the video data of the artificial agent will change from good to bad. In a specific embodiment of the present application, the above-mentioned emotion of the above-mentioned artificial agent is determined based on the above-mentioned image feature vector, the above-mentioned audio feature vector and the above-mentioned text feature vector, including: performing feature fusion based on the above-mentioned image feature vector, the above-mentioned audio feature vector and the above-mentioned text feature vector to obtain a first fused feature vector; comparing the above-mentioned first fused feature vector with multiple multi-dimensional emotion feature vectors to determine the first target feature vector with the smallest similarity to the above-mentioned first fused feature vector, and determining the emotion corresponding to the above-mentioned first target feature vector as the target emotion. In this embodiment, feature extraction can be performed based on video data, audio data and the first text, and based on the determined change trend of the state of the image feature vector, audio feature vector and text feature vector, the emotion of the artificial agent can be further efficiently and accurately determined by the range in which the first fused feature vector falls.

[0054] Step S205, determining the fatigue level of the artificial agent according to the image feature vector and the audio feature vector;

[0055] Changes in call status often do not occur instantaneously. Generally, there is a process for the call status to change from good to bad. During this process, the voice status corresponding to the first audio data of the artificial agent will change from good to bad. In a specific embodiment of the present application, the fatigue of the artificial agent is determined based on the image feature vector and the audio feature vector, including: performing feature fusion based on the image feature vector and the audio feature vector to obtain a second fused feature vector; comparing the second fused feature vector with a plurality of multi-dimensional fatigue feature vectors to determine the second target feature vector with the smallest similarity to the second fused feature vector, and determining the fatigue corresponding to the second target feature vector as the target fatigue. In this embodiment, feature extraction can be performed based on the first audio data and the first text, and the change trend can be determined based on the status of the image feature vector and the audio feature vector, and then the fatigue of the artificial agent can be further efficiently and accurately determined by the range in which the second fused feature vector falls.

[0056] Specifically, a neural network model may be used to determine the fatigue of the artificial agent. The second fused feature vector may be input into the neural network model, and the second fused feature vector may be compared with a plurality of multidimensional fatigue feature vectors through the neural network model to determine the fatigue degree of the artificial agent.

[0057] Step S206: determining the professionalism of the human agent according to the first text.

[0058] Changes in call status often do not occur instantaneously. Generally, there is a process for the call status to change from good to bad. During this process, the speech content corresponding to the first text of the manual agent will also change from good to bad. In a specific embodiment of the present application, the professionalism of the manual agent is determined based on the first text, including: extracting professional terms and non-professional terms in the first text; determining the professionalism of the manual agent based on a first proportion of the professional terms in the first text and a second proportion of the non-professional terms in the first text. In this embodiment, professional terms and non-professional terms can be extracted based on the first text, and the trend of change can be determined based on the proportion of professional terms and non-professional terms, and then the professionalism of the manual agent can be further determined efficiently and accurately through the proportion of professional terms and non-professional terms.

[0059] Specifically, the first text can be analyzed to extract professional terms and non-professional terms in the first text. Professional terms can include "query service", "your credit limit", etc. Non-professional terms refer to excessive saliva, such as "ah", "um", etc. Non-professional terms also include illegal terms and repetitive words. Of course, the professionalism of the artificial seat can also be determined based on the neural network model.

[0060] In an optional embodiment, when the first ratio is smaller than the second ratio, it can be determined that the professionalism of the artificial agent is low. When the first ratio is greater than a first threshold and the second ratio is smaller than a second threshold, it can be determined that the professionalism of the artificial agent is high and the first threshold is higher than the second threshold.

[0061] In the above-mentioned steps S201 to S206, multimodal state analysis can be performed on the conversation between the artificial agent and the user through video data, first audio data and first text, real-time analysis can be performed on the video data of the artificial agent, real-time analysis can be performed on the first text of the artificial agent, and multimodal detection and evaluation can be performed on the fatigue of the artificial agent to efficiently identify the emotional state, fatigue state and professionalism of the artificial agent, and then, based on the results of the multi-dimensional analysis, the virtual agent can be switched in time to continue communicating with the user.

[0062] In order to further perform real-time analysis on the second text corresponding to the user, and further efficiently and accurately determine the user's satisfaction, in another embodiment of the present application, determining the user's satisfaction according to the second text specifically includes the following steps:

[0063] Step S301, extracting target keywords from the second text, wherein the target keywords include at least one of the following: words indicating anger, words indicating dissatisfaction with the service, and words indicating the need to change an agent;

[0064] Step S302: determining the user's satisfaction level according to the number of the target keywords in the second text.

[0065] Specifically, a neural network model can be used to process the second text to analyze the number of target keywords in the second text. When the number is greater than a first quantity threshold, the user satisfaction is determined to be a first satisfaction. When the number is less than the first quantity, the user satisfaction is determined to be a second satisfaction. The first quantity threshold is less than the second quantity threshold, and the user satisfaction level of the first satisfaction level is higher than the user satisfaction level of the second satisfaction level.

[0066] In the above steps S301 to S302, target keywords can be extracted based on the second text, and the changing trend of user's service quality can be determined based on the number of target keywords, and the user's satisfaction can be further determined efficiently and accurately through the number of target keywords.

[0067] When the manual agent is temporarily unable to provide services to the user and before the next manual agent is forced to provide services to the user, the virtual agent can be used to take over the conversation in time, and it can be determined whether the conditions for switching to the virtual agent are met. In another embodiment of the present application, whether the switching conditions are met is determined according to the above-mentioned emotion of the above-mentioned manual agent, the above-mentioned fatigue of the above-mentioned manual agent, the above-mentioned professionalism of the above-mentioned manual agent and the above-mentioned satisfaction of the above-mentioned user, specifically including the following steps:

[0068] Step S401, obtaining a first score and a first weight coefficient corresponding to the emotion of the artificial agent, wherein the first score is determined according to the emotion of the artificial agent, and the first weight coefficient refers to the importance of the first score in the target score;

[0069] Step S402, obtaining a second score and a second weight coefficient corresponding to the fatigue level of the artificial seat, wherein the second score is determined according to the fatigue level of the artificial seat, and the second weight coefficient refers to the importance of the second score in the target score;

[0070] Step S403, obtaining a third score and a third weight coefficient corresponding to the professionalism of the artificial seat, wherein the third score is determined according to the professionalism of the artificial seat, and the third weight coefficient refers to the importance of the third score in the target score;

[0071] Step S404, obtaining a fourth score and a fourth weight coefficient corresponding to the satisfaction of the user, wherein the fourth score is determined according to the satisfaction of the user, and the fourth weight coefficient refers to the importance of the fourth score in the target score;

[0072] Specifically, in the process of general communication between agents and users, user satisfaction may be the most important, so the fourth weight coefficient may be greater than the first weight coefficient, the second weight coefficient, and the third weight coefficient.

[0073] Step S405, determining a target score according to the first score, the first weight coefficient, the second score, the second weight coefficient, the third score, the third weight coefficient, the fourth score, and the fourth weight coefficient;

[0074] In an optional embodiment, a first product of a first score and a first weight coefficient may be obtained, a second product of a second score and a second weight coefficient may be obtained, a third product of a third score and a third weight coefficient may be obtained, a fourth product of a fourth score and a fourth weight coefficient may be obtained, and then the sum of the first product, the second product, the third product and the fourth product may be obtained to obtain a target score.

[0075] Step S406: When the target score is greater than the score threshold, it is determined that the switching condition is met; when the target score is less than or equal to the score threshold, it is determined that the switching condition is not met.

[0076] Specifically, the scoring threshold may be 80 points or 85 points. Of course, it is not limited to these two cases, and may be any other feasible scoring threshold.

[0077] More specifically, when it is determined that the emotion of the human agent is negative, the fatigue of the human agent reaches the fatigue threshold, the language of the human agent does not meet professional requirements, or the user has doubts about the service quality or service capabilities of the human agent, it can be determined that the switching conditions are met and the virtual agent is controlled to take over the conversation, thereby converting the communication to a positive state.

[0078] In the above steps S401 to S406, based on the results of the multi-dimensional analysis of the call status, the manual agent can be switched to the virtual agent in time, and the virtual agent can be controlled to continue communicating with the user, so as to further realize the switching from the manual agent to the virtual agent.

[0079] In the case where it is necessary to switch the manual seat to the virtual seat, in order to prevent the user from suddenly discovering that the seat has been switched and to allow the virtual seat to continue talking to the user, the conversation process between the manual seat and the user can be monitored. In another embodiment of the present application, when it is determined that the above-mentioned switching condition is met, the above-mentioned manual seat is controlled to interrupt the conversation with the above-mentioned user, and the virtual seat is controlled to continue talking to the above-mentioned user, including:

[0080] Step S501, obtaining the timbre characteristics and voiceprint characteristics of the above-mentioned artificial seat;

[0081] Step S502, controlling the manual agent to interrupt the call with the user, and controlling the virtual agent to continue the call with the user based on the timbre feature and the voiceprint feature, wherein during the call between the manual agent and the user, the virtual agent monitors the call in real time.

[0082] Specifically, the conversation robot can obtain the interaction content between the human agent and the user in real time, and when the virtual agent is not switched, it can still generate the answer information that the user needs in real time, thereby ensuring that the virtual agent can track the progress of the interaction between the user and the human agent, and then ensure that the seamless switching between the human agent and the virtual agent will not be perceived by the user.

[0083] In the above steps S501 to S502, the virtual agent can take over the conversation with the user in real time. In subsequent conversations, the virtual agent uses a synthetic sound with the timbre characteristics and voiceprint characteristics of the artificial agent to conduct a scene conversation with the user to ensure that the virtual agent can take over the conversation at any time, and the user is unaware of the switch of the agent, thereby ensuring a good user experience.

[0084] In a specific embodiment, when it is determined that the switching conditions are met, a manual agent can also be prompted synchronously, the prompt content can be displayed on the display interface corresponding to the manual agent, and the audio playback device can be controlled to play a prompt voice to prompt the manual agent to adjust emotions, language and other states. After the manual agent is adjusted, the conversation with the user can be resumed, and the conversation content between the virtual agent and the user can be monitored during the adjustment of the manual agent.

[0085] The embodiment of the present application also provides a switching device for a customer service seat. It should be noted that the switching device for a customer service seat in the embodiment of the present application can be used to execute the switching method for a customer service seat provided in the embodiment of the present application. The switching device for a customer service seat provided in the embodiment of the present application is introduced below.

[0086] Figure 2 Schematic diagram of a switching device for a customer service agent according to an embodiment of the present application. Figure 2 As shown, the device comprises:

[0087] The first processing unit 10 is used to obtain the video data and the first audio data of the human agent during the speaking process, and convert the first audio data into a first text;

[0088] The above-mentioned first processing unit can obtain information of multiple dimensions, specifically, video data, first audio data and a first text generated based on the first audio data, so that the status of the artificial agent can be evaluated from multiple dimensions subsequently to further ensure the high accuracy of the status detection of the artificial agent.

[0089] The second processing unit 20 is used to obtain second audio data of the user during the speaking process, and convert the second audio data into a second text, wherein the user requests the artificial seat to provide artificial customer service services, and the artificial seat and the user talk;

[0090] The second processing unit can obtain user data from the user direction, so that the user satisfaction can be evaluated from the user dimension, and then it can be determined whether it is necessary to switch to a virtual seat to interact with the user according to the user satisfaction.

[0091] A determination unit 30, configured to determine the emotion, fatigue and professionalism of the artificial agent according to the video data, the first audio data and the first text, and determine the user's satisfaction according to the second text;

[0092] The above-mentioned determination unit can determine the change trend of the artificial seat service based on the multimodal data of the video data, the first audio data and the first text, and can predict the call status. When the call status deviates from good to bad, a warning can be issued in time, and when the switching conditions are met, the virtual seat can take over the conversation in time to interact with the user. That is, the data between the artificial seat and the user can be collected, and the call status can be evaluated from multiple dimensions of both the artificial seat and the user. In this way, the evaluation dimensions are richer and more complete, which can ensure more accurate subsequent switching.

[0093] The third processing unit 40 is used to determine whether the switching condition is met according to the above-mentioned emotion of the above-mentioned manual agent, the above-mentioned fatigue of the above-mentioned manual agent, the above-mentioned professionalism of the above-mentioned manual agent and the above-mentioned satisfaction of the above-mentioned user, and when it is determined that the above-mentioned switching condition is met, control the above-mentioned manual agent to interrupt the call with the above-mentioned user, and control the virtual agent to continue the call with the above-mentioned user, wherein the above-mentioned virtual agent refers to a virtual character generated by artificial intelligence, which provides virtual customer service services to the above-mentioned user through the chat robot function.

[0094] The third processing unit mentioned above can switch from a human agent to a virtual agent in a timely manner through the results of a multi-dimensional analysis of the call status, and can also control the virtual agent to use the synthetic voice of the human agent to communicate with the user in a good emotional and language state, so that the user can be less aware of the human-machine switch as much as possible, thereby ensuring a good user experience. By using the virtual agent as a backup for the human agent, the key mitigation of user service quality assurance can be supplemented, thereby further improving the call efficiency between the customer service agent and the user, and ensuring that the human agent is always in the best state to communicate with the user, which can reduce the complaint rate of the human agent.

[0095] In the above-mentioned device, the first processing unit obtains the video data and the first audio data of the artificial agent during the speaking process, and converts the first audio data into the first text, the second processing unit obtains the second audio data of the user during the speaking process, and converts the second audio data into the second text, the determination unit determines the emotion, fatigue and professionalism of the artificial agent according to the video data, the first audio data and the first text, and determines the user's satisfaction according to the second text, and the third processing unit determines whether the switching condition is met according to the emotion of the artificial agent, the fatigue of the artificial agent, the professionalism of the artificial agent and the user's satisfaction, and when it is determined that the switching condition is met, the artificial agent is controlled to interrupt the call with the user, and the virtual agent is controlled to continue the call with the user. In this solution, based on the combination of multimodal emotion detection, fatigue detection, professionalism detection and text analysis technology, the interaction status between the artificial agent and the user can be analyzed from multiple dimensions, and the scene dialogue of the virtual agent can be seamlessly connected according to the analysis results, so as to realize accurate automatic switching between virtual agents and artificial agents.

[0096] In order to ensure the accuracy of the detection of the emotion, fatigue and professionalism of the artificial seat, the emotion, fatigue and professionalism of the artificial seat can be detected by feature extraction and feature vector. In one embodiment of the present application, the determination unit includes a first extraction module, a second extraction module, a third extraction module, a first determination module, a second determination module and a third determination module. The functions of each module are as follows:

[0097] A first extraction module is used to extract features from the video data to obtain an image feature vector;

[0098] A second extraction module, used to extract features from the first audio data to obtain an audio feature vector;

[0099] A third extraction module is used to extract features from the first text to obtain a text feature vector;

[0100] A first determination module, configured to determine the emotion of the artificial agent according to the image feature vector, the audio feature vector and the text feature vector;

[0101] The change of call status often does not happen instantaneously. Generally, there is a process for the call status to change from good to bad. During this process, the facial expression corresponding to the video data of the artificial seat will change from good to bad. In a specific embodiment of the present application, the first determination module includes a first fusion submodule and a first determination submodule. The first fusion submodule is used to perform feature fusion according to the above-mentioned image feature vector, the above-mentioned audio feature vector and the above-mentioned text feature vector to obtain a first fused feature vector; the first determination submodule is used to compare the above-mentioned first fused feature vector with multiple multi-dimensional emotion feature vectors, determine the first target feature vector with the smallest similarity to the above-mentioned first fused feature vector, and determine the emotion corresponding to the above-mentioned first target feature vector as the target emotion. In this embodiment, feature extraction can be performed based on video data, audio data and the first text, and based on the determined change trend of the state of the image feature vector, audio feature vector and text feature vector, the emotion of the artificial seat can be further efficiently and accurately determined by the range in which the first fused feature vector falls.

[0102] A second determination module, configured to determine the fatigue level of the artificial agent according to the image feature vector and the audio feature vector;

[0103] Changes in call status often do not occur instantaneously. Generally, there is a process for the call status to change from good to bad. During this process, the voice status corresponding to the first audio data of the artificial seat will change from good to bad. In a specific embodiment of the present application, the second determination module includes a second fusion submodule and a second determination submodule. The second fusion submodule is used to perform feature fusion based on the above-mentioned image feature vector and the above-mentioned audio feature vector to obtain a second fused feature vector; the second determination submodule is used to compare the above-mentioned second fused feature vector with multiple multi-dimensional fatigue feature vectors, determine the second target feature vector with the smallest similarity to the above-mentioned second fused feature vector, and determine the fatigue corresponding to the above-mentioned second target feature vector as the target fatigue. In this embodiment, feature extraction can be performed based on the first audio data and the first text, and the change trend can be determined based on the state of the image feature vector and the audio feature vector, and then the fatigue of the artificial seat can be further efficiently and accurately determined by the range in which the second fused feature vector falls.

[0104] The third determination module is used to determine the professionalism of the artificial agent according to the first text.

[0105] The change of call status often does not happen instantaneously. Generally, there is a process for the call status to change from good to bad. During this process, the speech content corresponding to the first text of the manual agent will change from good to bad. In a specific embodiment of the present application, the third determination module includes an extraction submodule and a third determination submodule. The extraction submodule is used to extract professional terms and non-professional terms in the above-mentioned first text; the third determination submodule is used to determine the above-mentioned professionalism of the above-mentioned manual agent according to the first proportion of the above-mentioned professional terms in the above-mentioned first text and the second proportion of the above-mentioned non-professional terms in the above-mentioned first text. In this embodiment, professional terms and non-professional terms can be extracted based on the first text, and the change trend can be determined based on the proportion of professional terms and non-professional terms, and then the professionalism of the manual agent can be further determined efficiently and accurately through the proportion of professional terms and non-professional terms.

[0106] The above-mentioned first extraction module, second extraction module, third extraction module, first determination module, second determination module and third determination module can perform multimodal state analysis on the conversation between the artificial agent and the user through video data, first audio data and first text, can perform real-time analysis on the video data of the artificial agent, can perform real-time analysis on the first text of the artificial agent, can perform multimodal detection and evaluation on the fatigue of the artificial agent, so as to efficiently identify the emotional state, fatigue state and professionalism of the artificial agent, and then through the results of multi-dimensional analysis, the virtual agent can be switched in time to continue communicating with the user.

[0107] In order to further perform real-time analysis on the second text corresponding to the user, and further efficiently and accurately determine the user's satisfaction, in another embodiment of the present application, the determination unit includes a fourth extraction module and a fourth determination module, and the functions of each module are as follows:

[0108] A fourth extraction module is used to extract target keywords from the second text, wherein the target keywords include at least one of the following: words representing anger, words representing dissatisfaction with the service, and words representing the need to change a human agent;

[0109] The fourth determination module is used to determine the user's satisfaction level according to the number of the target keywords in the second text.

[0110] The fourth extraction module and the fourth determination module can extract target keywords based on the second text, and determine the changing trend of user's service quality based on the number of target keywords, and further determine the user's satisfaction efficiently and accurately through the number of target keywords.

[0111] When the manual agent is temporarily unable to provide services to the user and before forcibly switching to the next manual agent to provide services to the user, the virtual agent can be used to take over the conversation in time, and it can be determined whether the conditions for switching to the virtual agent are met. In another embodiment of the present application, the third processing unit includes a first acquisition module, a second acquisition module, a third acquisition module, a fourth acquisition module, a fifth determination module and a sixth determination module, and the functions of each module are as follows:

[0112] a first acquisition module, configured to acquire a first score and a first weight coefficient corresponding to the emotion of the artificial agent, wherein the first score is determined according to the emotion of the artificial agent, and the first weight coefficient refers to the importance of the first score in the target score;

[0113] A second acquisition module is used to acquire a second score and a second weight coefficient corresponding to the fatigue level of the artificial seat, wherein the second score is determined according to the fatigue level of the artificial seat, and the second weight coefficient refers to the importance of the second score in the target score;

[0114] a third acquisition module, configured to acquire a third score and a third weight coefficient corresponding to the professionalism of the artificial agent, wherein the third score is determined according to the professionalism of the artificial agent, and the third weight coefficient refers to the importance of the third score in the target score;

[0115] a fourth acquisition module, configured to acquire a fourth score and a fourth weight coefficient corresponding to the satisfaction of the user, wherein the fourth score is determined according to the satisfaction of the user, and the fourth weight coefficient refers to the importance of the fourth score in the target score;

[0116] a fifth determination module, configured to determine a target score according to the first score, the first weight coefficient, the second score, the second weight coefficient, the third score, the third weight coefficient, the fourth score, and the fourth weight coefficient;

[0117] The sixth determination module is used to determine that the switching condition is met when the target score is greater than the score threshold, and to determine that the switching condition is not met when the target score is less than or equal to the score threshold.

[0118] The first acquisition module, the second acquisition module, the third acquisition module, the fourth acquisition module, the fifth determination module and the sixth determination module can timely switch from a manual agent to a virtual agent through the results of a multi-dimensional analysis of the call status, and control the virtual agent to continue communicating with the user, so as to further realize the switching from a manual agent to a virtual agent.

[0119] In the case where it is necessary to switch the manual seat to the virtual seat, in order to prevent the user from suddenly discovering that the seat has been switched and the virtual seat can continue to talk to the user, the conversation process between the manual seat and the user can be monitored. In another embodiment of the present application, the third processing unit includes a fifth acquisition module and a control module, and the functions of each module are as follows:

[0120] A fifth acquisition module, used to acquire the timbre characteristics and voiceprint characteristics of the above-mentioned artificial seat;

[0121] The control module is used to control the above-mentioned manual agent to interrupt the call with the above-mentioned user, and control the above-mentioned virtual agent to continue the call with the above-mentioned user in combination with the above-mentioned timbre characteristics and the above-mentioned voiceprint characteristics, wherein, during the process of the above-mentioned manual agent talking with the above-mentioned user, the above-mentioned virtual agent monitors the process of the call in real time.

[0122] The fifth acquisition module and control module mentioned above can be used by the virtual agent to take over the conversation with the user in real time. In subsequent conversations, the virtual agent uses the synthesized sound of the timbre characteristics and voiceprint characteristics of the artificial agent to conduct scene conversations with the user to ensure that the virtual agent can take over the conversation at any time, and the user is unaware of the switch of the agent, thereby ensuring a better user experience.

[0123] The switching device for the customer service seat includes a processor and a memory. The first processing unit, the second processing unit, the determination unit and the third processing unit are all stored in the memory as program units. The processor executes the program units stored in the memory to realize corresponding functions.

[0124] The processor contains a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the manual seat can be switched to the virtual seat by adjusting the kernel parameters.

[0125] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0126] An embodiment of the present invention provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned customer service agent switching method is implemented.

[0127] An embodiment of the present invention provides a processor, and the processor is used to run a program, wherein the method for switching customer service seats is executed when the program is running.

[0128] The present application also provides an electronic device, comprising one or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing any one of the above methods.

[0129] This application also provides a customer service agent switching system, such as Figure 3 As shown, it includes a call center module, a comprehensive analysis module, a dialogue robot module, users, manual seats and virtual seats; wherein the call center module is used to manage the call connection and switching between users and manual seats and virtual seats; the comprehensive analysis module is used to provide real-time feedback to the call center module on whether to conduct a conclusion on manual seats and virtual seats based on the collected call data between users and manual seats (including: manual seat video data and first audio data, user's second audio data) and based on a comprehensive analysis of multi-dimensional call status; the dialogue robot module is used for virtual seats based on intelligent voice interaction, and can take over the dialogue interaction with users at any time according to the switching instructions of the call center module.

[0130] Specifically, the processing flow of the customer service agent switching system is as follows: Figure 4 As shown, the process is as follows:

[0131] Step 1: For multimodal emotion detection of human agents, the audio and video acquisition module collects the call data of human agents in real time through cameras and microphones. The video data such as facial expressions and facial blood oxygen saturation of human agents obtained by the camera are sent to the image feature model to obtain image feature vectors. The first audio data of the human agent obtained by the microphone is sent to the audio feature model to obtain audio feature vectors. The first text transcribed in real time by voice recognition is sent to the text feature model to obtain text feature vectors. The image feature vector, audio feature vector and text feature vector are sent to the multimodal emotion detection submodule. These types of feature data are sent to the feature vector space of the emotion feature classification model respectively to compare the multidimensional feature vectors, so as to obtain the closest emotion feature type and confidence score.

[0132] Step 2: Send the image feature vector and audio feature vector of the human agent to the multimodal fatigue detection submodule for multi-dimensional feature vector comparison to identify the fatigue level and score of the human agent;

[0133] Step 3: Perform speech recognition on the first voice data of the human agent and transcribe it into a first text. Then, the transcribed first text is sent to the text analysis model of the agent call content analysis submodule to determine the presence of non-professional language (for example, too much saliva such as "ah", "um", "that", etc., illegal language is detected, too much repetitive language, etc.), and the professionalism and score of the human agent are obtained according to the text analysis model;

[0134] Step 4: The audio collection module collects user call data in real time through a microphone, performs voice recognition on the user's second voice data, and transcribes it into a second text. The transcribed second text is then sent to the text analysis model of the user call content analysis submodule to determine the occurrence of user dissatisfied language (for example, the occurrence of keywords such as insults, dissatisfaction, and request for replacement), and the user satisfaction status and score are obtained based on the text analysis model.

[0135] Step 5: The multi-dimensional analysis conclusion type, degree and score output by the above sub-modules are simultaneously input into the call status evaluation model of the "multi-dimensional call status comprehensive analysis module", and the final judgment score is obtained according to the score of the call status evaluation model. When the score value exceeds the specified threshold, it is determined that it is necessary to switch to the virtual seat, and the instruction is passed to the call center module;

[0136] Step 6: The call center module switches to the virtual agent, and the virtual agent takes over the conversation with the user in real time. The subsequent conversation is carried out by the virtual agent using the synthetic voice of the human agent to conduct a scene dialogue with the user and call the intelligent question and answer of the standard knowledge base to continue; (When the user and the human agent start the conversation, the conversation robot module corresponding to the virtual agent starts to monitor the conversation between the user and the human agent in real time throughout the whole process, and outputs the corresponding response results according to each sentence of the user's conversation, so as to ensure that the virtual agent can take over the conversation at any time);

[0137] Step 7: The system will simultaneously prompt the human agent to adjust his / her emotions, language, and other status in a timely manner. When the system determines that the status of the human agent and the user has returned to normal, the conversation will be switched to the human agent again.

[0138] An embodiment of the present invention provides a device, the device comprising a processor, a memory, and a program stored in the memory and executable on the processor, and when the processor executes the program, at least the following steps are implemented:

[0139] Step S101, obtaining video data and first audio data of a human agent during a speaking process, and converting the first audio data into a first text;

[0140] Step S102, obtaining second audio data of the user during the speaking process, and converting the second audio data into a second text, wherein the user requests the artificial agent to provide artificial customer service services, and the artificial agent and the user have a conversation;

[0141] Step S103, determining the emotion, fatigue and professionalism of the artificial agent according to the video data, the first audio data and the first text, and determining the user's satisfaction according to the second text;

[0142] Step S104, determining whether the switching condition is met according to the emotion of the artificial agent, the fatigue of the artificial agent, the professionalism of the artificial agent and the satisfaction of the user; if it is determined that the switching condition is met, controlling the artificial agent to interrupt the call with the user, and controlling the virtual agent to continue the call with the user, wherein the virtual agent refers to a virtual character generated by artificial intelligence, which provides virtual customer service services to the user through a chatbot function.

[0143] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0144] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing at least the following method steps:

[0145] Step S101, obtaining video data and first audio data of a human agent during a speaking process, and converting the first audio data into a first text;

[0146] Step S102, obtaining second audio data of the user during the speaking process, and converting the second audio data into a second text, wherein the user requests the artificial agent to provide artificial customer service services, and the artificial agent and the user have a conversation;

[0147] Step S103, determining the emotion, fatigue and professionalism of the artificial agent according to the video data, the first audio data and the first text, and determining the user's satisfaction according to the second text;

[0148] Step S104, determining whether the switching condition is met according to the emotion of the artificial agent, the fatigue of the artificial agent, the professionalism of the artificial agent and the satisfaction of the user; if it is determined that the switching condition is met, controlling the artificial agent to interrupt the call with the user, and controlling the virtual agent to continue the call with the user, wherein the virtual agent refers to a virtual character generated by artificial intelligence, which provides virtual customer service services to the user through a chatbot function.

[0149] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0150] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the above-mentioned units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0151] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0152] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0153] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the above-mentioned methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0154] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0155] 1) The method for switching customer service seats of the present application first obtains the video data and first audio data of the artificial agent during the speaking process, and converts the first audio data into a first text, then obtains the second audio data of the user during the speaking process, and converts the second audio data into a second text, then determines the emotion, fatigue and professionalism of the artificial agent based on the video data, the first audio data and the first text, determines the user's satisfaction based on the second text, and finally determines whether the switching conditions are met based on the emotion of the artificial agent, the fatigue of the artificial agent, the professionalism of the artificial agent and the user's satisfaction. When it is determined that the switching conditions are met, the artificial agent is controlled to interrupt the call with the user, and the virtual agent is controlled to continue the call with the user. In this solution, based on the combination of multimodal emotion detection, fatigue detection, professionalism detection and text analysis technology, the interaction status between the artificial agent and the user can be analyzed from multiple dimensions, and the scene dialogue of the virtual agent can be seamlessly connected based on the analysis results, thereby realizing accurate automatic switching between virtual agents and artificial agents.

[0156] 2) The switching device for customer service seats of the present application, the first processing unit obtains the video data and the first audio data of the artificial agent during the speaking process, and converts the first audio data into the first text, the second processing unit obtains the second audio data of the user during the speaking process, and converts the second audio data into the second text, the determination unit determines the emotion, fatigue and professionalism of the artificial agent according to the video data, the first audio data and the first text, and determines the user's satisfaction according to the second text, and the third processing unit determines whether the switching conditions are met according to the emotion of the artificial agent, the fatigue of the artificial agent, the professionalism of the artificial agent and the user's satisfaction, and when it is determined that the switching conditions are met, the artificial agent is controlled to interrupt the call with the user, and the virtual agent is controlled to continue the call with the user. In this scheme, based on the combination of multimodal emotion detection, fatigue detection, professionalism detection and text analysis technology, the interaction status between the artificial agent and the user can be analyzed from multiple dimensions, and the scene dialogue of the virtual agent can be seamlessly connected according to the analysis results, so as to realize accurate automatic switching between virtual agents and artificial agents.

[0157] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for switching a customer service seat, characterized in that: include: Acquire video data and first audio data of the human agent during speaking, and convert the first audio data into a first text; Acquire second audio data of the user during the speaking process, and convert the second audio data into a second text, wherein the user requests the artificial agent to provide artificial customer service services, and the artificial agent and the user have a conversation; Determine the emotion, fatigue and professionalism of the human agent according to the video data, the first audio data and the first text, and determine the user's satisfaction according to the second text; determining whether a switching condition is met according to the emotion of the human agent, the fatigue of the human agent, the professionalism of the human agent and the satisfaction of the user, and if it is determined that the switching condition is met, controlling the human agent to interrupt the call with the user and controlling the virtual agent to continue the call with the user, wherein the virtual agent refers to a virtual character generated by artificial intelligence, and provides virtual customer service services to the user through a chatbot function; When it is determined that the switching condition is met, controlling the manual agent to interrupt the call with the user, and controlling the virtual agent to continue the call with the user, including: Acquiring the timbre characteristics and voiceprint characteristics of the artificial agent; The manual agent is controlled to interrupt the call with the user, and the virtual agent is controlled to continue the call with the user in combination with the timbre feature and the voiceprint feature, wherein during the call between the manual agent and the user, the virtual agent monitors the call in real time.

2. The method according to claim 1, characterized in that Determining the emotion, fatigue and professionalism of the human agent according to the video data, the first audio data and the first text includes: Extracting features from the video data to obtain an image feature vector; Performing feature extraction on the first audio data to obtain an audio feature vector; Performing feature extraction on the first text to obtain a text feature vector; Determine the emotion of the human agent according to the image feature vector, the audio feature vector and the text feature vector; Determining the fatigue degree of the human agent according to the image feature vector and the audio feature vector; The professionalism of the human agent is determined according to the first text.

3. The method according to claim 2, characterized in that Determining the emotion of the artificial agent according to the image feature vector, the audio feature vector, and the text feature vector includes: Perform feature fusion according to the image feature vector, the audio feature vector and the text feature vector to obtain a first fused feature vector; Comparing the first fused feature vector with a plurality of multi-dimensional emotion feature vectors, determining a first target feature vector having the smallest similarity to the first fused feature vector, and determining the emotion corresponding to the first target feature vector as the target emotion; Determining the fatigue degree of the artificial agent according to the image feature vector and the audio feature vector includes: Perform feature fusion according to the image feature vector and the audio feature vector to obtain a second fused feature vector; The second fused feature vector is compared with a plurality of multi-dimensional fatigue feature vectors to determine a second target feature vector having the smallest similarity to the second fused feature vector, and the fatigue degree corresponding to the second target feature vector is determined as a target fatigue degree.

4. The method according to claim 2, characterized in that: Determining the professionalism of the human agent according to the first text includes: Extracting professional words and non-professional words from the first text; The professionalism of the human agent is determined according to a first proportion of the professional words in the first text and a second proportion of the non-professional words in the first text.

5. The method according to claim 1, characterized in that Determining the user's satisfaction level according to the second text includes: Extracting target keywords from the second text, wherein the target keywords include at least one of the following: words indicating anger, words indicating dissatisfaction with the service, and words indicating the need to change an agent; The user's satisfaction is determined according to the number of the target keywords in the second text.

6. The method according to claim 1, characterized in that Determining whether a switching condition is met according to the emotion of the artificial agent, the fatigue of the artificial agent, the professionalism of the artificial agent, and the satisfaction of the user includes: Obtaining a first score and a first weight coefficient corresponding to the emotion of the artificial agent, wherein the first score is determined according to the emotion of the artificial agent, and the first weight coefficient refers to the importance of the first score in the target score; Obtaining a second score and a second weight coefficient corresponding to the fatigue level of the human agent, where the second score is determined according to the fatigue level of the human agent, and the second weight coefficient refers to the importance of the second score in the target score; Obtaining a third score and a third weight coefficient corresponding to the professionalism of the human agent, wherein the third score is determined according to the professionalism of the human agent, and the third weight coefficient refers to the importance of the third score in the target score; Obtaining a fourth score and a fourth weight coefficient corresponding to the satisfaction of the user, wherein the fourth score is determined according to the satisfaction of the user, and the fourth weight coefficient refers to the importance of the fourth score in the target score; Determine a target score according to the first score, the first weight coefficient, the second score, the second weight coefficient, the third score, the third weight coefficient, the fourth score, and the fourth weight coefficient; When the target score is greater than the score threshold, it is determined that the switching condition is met, and when the target score is less than or equal to the score threshold, it is determined that the switching condition is not met.

7. A switching device for customer service seats, characterized in that: include: A first processing unit, configured to obtain video data and first audio data of the human agent during the speaking process, and convert the first audio data into a first text; A second processing unit is used to obtain second audio data of the user during the speaking process, and convert the second audio data into a second text, wherein the user requests the artificial seat to provide artificial customer service services, and the artificial seat and the user talk; a determination unit, configured to determine the emotion, fatigue and professionalism of the human agent according to the video data, the first audio data and the first text, and determine the user's satisfaction according to the second text; a third processing unit, configured to determine whether a switching condition is met according to the emotion of the human agent, the fatigue of the human agent, the professionalism of the human agent, and the satisfaction of the user, and, if it is determined that the switching condition is met, control the human agent to interrupt the call with the user, and control the virtual agent to continue the call with the user, wherein the virtual agent refers to a virtual character generated by artificial intelligence, and provides virtual customer service services to the user through a chatbot function; The third processing unit includes a fifth acquisition module and a control module, the fifth acquisition module is used to obtain the timbre characteristics and voiceprint characteristics of the artificial agent; the control module is used to control the artificial agent to interrupt the call with the user, and control the virtual agent to continue the call with the user in combination with the timbre characteristics and the voiceprint characteristics, wherein, during the call between the artificial agent and the user, the virtual agent monitors the call in real time.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method of any one of claims 1 to 6.

9. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of claims 1 to 6.

Images