Customer service method and customer service robot for real-time emotion

Abstract

The invention discloses a customer service method and a customer service robot for real-time emotion. The method comprises the following steps: receiving multi-modal input information input by a customer; semantic information and emotion information are extracted from the multi-modal input information; determining inquiry keywords according to the semantic information; inquiring a knowledge base according to the inquiry keyword so as to determine response information; determining a real-time emotion label according to the emotion information; determining a reply scheme according to the response information and the real-time emotion label; the semantic information and the emotion information are determined through the multi-modal input information, the customer service is performed in combination with the real-time emotion of the customer, and the reply scheme for the specific emotion is determined, so that extra emotion value can be provided and the customer experience can be improved while the problem can be solved for the customer.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a customer service method and a customer service robot targeting real-time emotions. Background Art

[0002] Currently, customer service has largely transitioned from traditional manual methods to a human-machine approach. This involves using a "customer service robot" capable of natural language conversation to answer user questions. For example, many large shopping malls and communities have deployed these robots in public areas, providing assistance and guidance to customers and, in some cases, completing tasks such as delivering goods.

[0003] However, many customers are currently dissatisfied with the capabilities of customer service robots. Natural language question-and-answer services primarily rely on rule-based engines (such as keyword matching and decision trees), using predefined rules to handle simple, "question-and-answer" inquiries. These robots lack the ability to handle complex scenarios, contextual understanding, and limited functionality. In other words, they can only respond to fixed, simple customer questions and are unable to adapt to complex functional requirements. For example, they are unable to adapt to a customer's current mood and state. Summary of the Invention

[0004] The present invention provides a customer service method and a customer service robot targeting real-time emotions, which can provide different response plans based on the real-time emotions of customers.

[0005] In a first aspect, the present invention provides a customer service method targeting real-time emotions, the method being applied to a customer service robot, comprising:

[0006] Receiving multimodal input information input by a customer; extracting semantic information and emotional information from the multimodal input information;

[0007] Determining query keywords based on the semantic information; and querying a knowledge base based on the query keywords to determine response information;

[0008] Determining a real-time emotion tag based on the emotion information;

[0009] A reply plan is determined according to the response information and the real-time emotion tag.

[0010] Preferably, the multimodal information includes content information and biometric information; the multimodal input information received from the customer includes:

[0011] Receiving the content information using an input interface;

[0012] Use sensing components to collect customers' biometric information.

[0013] Preferably, the extracting semantic information and emotional information from the multimodal input information includes:

[0014] Extracting the semantic information from the content information;

[0015] The emotion information is extracted from the biometric information.

[0016] Preferably, the biometric information includes expression information, action information and / or tone information; and determining the real-time emotion tag based on the emotion information includes:

[0017] Performing emotion analysis on the facial expression information, the action information, and / or the tone information using an analysis network to determine a corresponding emotion type;

[0018] The real-time emotion tag is determined according to the emotion type.

[0019] Preferably, determining a reply plan based on the response information and the real-time emotion tag includes:

[0020] Determining a corresponding emotional response plan based on the real-time emotional label;

[0021] The reply plan is determined according to the response information and the emotional coping plan.

[0022] In a second aspect, the present invention provides a customer service robot, comprising:

[0023] A sensing component for collecting biometric information;

[0024] Input interface, used to receive content information;

[0025] The processor component is used to extract semantic information and emotional information; determine query keywords based on the semantic information; and query the knowledge base based on the query keywords to determine response information; determine real-time emotional tags based on the emotional information; and determine a reply plan based on the response information and the real-time emotional tags.

[0026] Preferably, it also includes:

[0027] The main body structure includes an omnidirectional wheel chassis, a laser radar, a robotic arm, a touch display screen and a speaker.

[0028] In a third aspect, the present invention provides a customer service device for real-time emotions, characterized in that the device is placed in a customer service robot and includes:

[0029] A multimodal information receiving module, configured to receive multimodal input information input by a customer; and extract semantic information and emotional information from the multimodal input information;

[0030] A response information determination module is used to determine the query keyword based on the semantic information; and query the knowledge base based on the query keyword to determine the response information;

[0031] A sentiment analysis module, configured to determine a real-time sentiment tag based on the sentiment information;

[0032] The response plan determination module is used to determine a response plan based on the response information and the real-time emotion tag.

[0033] In a fourth aspect, the present invention provides a readable medium comprising an execution instruction. When a processor of an electronic device executes the execution instruction, the electronic device executes any method described in the first aspect.

[0034] In a fifth aspect, the present invention provides an electronic device comprising a processor and a memory storing execution instructions, wherein when the processor executes the execution instructions stored in the memory, the processor executes any method described in the first aspect.

[0035] The present invention provides a customer service method and a customer service robot targeting real-time emotions. The method determines semantic information and emotional information through multimodal input information, provides customer service in combination with the customer's real-time emotions, and determines response plans for specific emotions. In addition to solving problems for customers, the method can provide additional emotional value and improve customer experience.

[0036] The further effects of the above-mentioned non-conventional preferred embodiment will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the existing technical solutions, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 A flowchart of a customer service method for real-time emotions provided by one embodiment of the present invention;

[0039] Figure 2 A flowchart of another customer service method for real-time emotions provided by an embodiment of the present invention;

[0040] Figure 3A schematic structural diagram of a customer service robot provided by one embodiment of the present invention;

[0041] Figure 4 A schematic structural diagram of a customer service device for real-time emotions provided by one embodiment of the present invention;

[0042] Figure 5 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0044] Currently, customer service has largely transitioned from traditional manual methods to a human-machine approach. This involves using a "customer service robot" capable of natural language conversation to answer user questions. For example, many large shopping malls and communities have deployed these robots in public areas, providing assistance and guidance to customers and, in some cases, completing tasks such as delivering goods.

[0045] However, many customers are currently dissatisfied with the capabilities of customer service robots. Natural language question-and-answer services primarily rely on rule-based engines (such as keyword matching and decision trees), using predefined rules to handle simple, "question-and-answer" inquiries. These robots lack the ability to handle complex scenarios, contextual understanding, and limited functionality. In other words, they can only respond to fixed, simple customer questions and are unable to adapt to complex functional requirements. For example, they are unable to adapt to a customer's current mood and state.

[0046] In view of this, the present invention provides a customer service method targeting real-time emotions. Figure 1 As shown in FIG, a specific embodiment of the customer service method for real-time emotions provided by the present invention is provided. The method is applied to a customer service robot. In this embodiment, the method includes:

[0047] Step 101: Receive multimodal input information input by a customer; and extract semantic information and emotional information from the multimodal input information.

[0048] In this embodiment, the customer service robot's sensor component / input interface (or perception layer) is used to receive multimodal information. The multimodal information includes content information and biometric information. Receiving multimodal input from the customer includes: receiving the content information via the input interface; and collecting the customer's biometric information via the sensor component.

[0049] Specifically, the sensing components in the perception layer may include: RGB-D cameras, microphone arrays, infrared thermal imagers, environmental sensors, and more. RGB-D cameras are used for facial recognition, 3D modeling, and micro-expression capture. Microphone arrays are used for sound source localization, noise reduction, and voiceprint extraction. Infrared thermal imagers are used for gait analysis, temperature monitoring, and night vision. All of the data obtained above constitutes biometric information, which refers to certain physiological characteristics displayed by the customer at the moment.

[0050] The input interface refers to the interface through which the customer service robot interacts with customers. It receives information actively input by customers, namely, content information. For example, inquiries. Content information can be text, voice, or other types of information. The input interface is the channel through which customers interact with the customer service robot.

[0051] It's understandable that the aforementioned biometric information reflects certain "states" exhibited by the customer at that moment, and this "state" can clearly reflect the customer's "emotions" to a certain extent. For example, facial micro-expressions, body movements, speech speed and tone, etc. can all reflect emotions and can be extracted from the biometric information collected by the aforementioned sensor components. In this embodiment, all types of information extracted from biometric information that represents customer emotions are collectively referred to as emotional information.

[0052] On the other hand, the content information entered by customers through the input interface will include specific inquiries, which represent their demands, or in other words, include the problems they hope to solve through customer service. In customer service, the specific meaning of this demand also needs to be analyzed.

[0053] That is, in this embodiment, the semantic information is extracted from the content information; and the emotion information is extracted from the biometric information.

[0054] Step 102: Determine query keywords based on the semantic information; and query a knowledge base based on the query keywords to determine response information.

[0055] The steps in this embodiment are similar to those of traditional customer service. That is, the query keywords are determined based on the semantic information and the specific demands are analyzed. Then, the relevant knowledge base is queried based on the query keywords to determine the response information. The response information is the answer to the "demand" or "question". In normal customer service, after receiving the response information, it can be directly fed back to the customer to answer his question. In other words, the simple and fixed-pattern question-and-answer process in the prior art can end here. However, in this embodiment, the process will be further improved and optimized through the following steps.

[0056] Step 103: Determine a real-time emotion tag based on the emotion information.

[0057] In this embodiment, in addition to providing conventional "answers" to customers, it can also further identify the customer's current emotions, that is, real-time emotions, and thus adaptively adjust the "answer" method based on emotions. In this way, in addition to answering the customer's "questions", it can also further provide a certain degree of "emotional value" to the customer, thereby improving the experience.

[0058] Combined with the aforementioned emotional information, targeted emotional analysis can be performed on the customer. Specifically, the biometric information includes facial expression information, action information, and / or voice tone information; determining a real-time emotional tag based on the emotional information includes: utilizing an analysis network to perform emotional analysis on the facial expression information, action information, and / or voice tone information to determine a corresponding emotional type; and determining the real-time emotional tag based on the emotional type.

[0059] Specifically, the biometric fusion recognition engine can be used to determine real-time emotion tags. The specific functions of the biometric fusion recognition engine are:

[0060] Face recognition, based on the ArcFace algorithm, achieved high accuracy on the LFW dataset. Voiceprint recognition: Utilizes the x-vector + PLDA model and supports cross-channel verification. Gait recognition: Analyzes spatiotemporal features through a 3D convolutional neural network (3D-CNN) to address occlusion scenarios. Micro-expression recognition: Identifies FACS action units (AUs) based on a deep residual network (ResNet-50). Speech sentiment analysis uses the Wav2Vec 2.0 pre-trained model, fine-tuned on the IEMOCAP sentiment corpus. Multimodal sentiment fusion analysis utilizes the Transformer architecture and combines a temporal alignment algorithm to synchronize speech and visual features. Furthermore, it can also provide interference-resistant biometric recognition capabilities. This involves integrating multimodal information from face, voiceprint, gait, and micro-expression to address the challenge of recognizing a single feature in the presence of occlusion (such as masks and sunglasses) or in noisy environments. For example, when a person's face is obscured, the system automatically switches to voiceprint or gait verification.

[0061] The final real-time emotion tag can indicate a single emotion of the customer, such as happiness, anger, panic, etc. It can also be a complex tag involving multiple emotions and different levels of emotion.

[0062] Step 104: Determine a reply plan based on the response information and the real-time emotion tag.

[0063] Combining the aforementioned response information with real-time emotional tags, a comprehensive response plan can be determined. This is the specific content that will be provided to the customer. As mentioned above, a response is a simple "answer" devoid of emotion. For example, when a user asks a simple question like, "How long will the queue last?" the response, devoid of emotion, often simply provides a time, such as "30 minutes." However, in reality, when a customer asks the same question under different emotional conditions, their mental state and desired response can be completely different. If a customer is happy, they may enjoy the excitement of the moment. In this case, the response plan can include more relevant information and interactive elements to help them fully experience the experience. Conversely, if a customer is anxious, perhaps tired of the long wait, the response plan can include guidance and comfort, or offer some entertainment to help them pass the tedious wait.

[0064] It can be seen that, in addition to being able to "answer questions", the response scheme in this embodiment can also provide additional emotional value by combining customer emotions, which obviously improves the customer experience.

[0065] It can be seen from the above technical solutions that this embodiment has at least the following beneficial effects: determining semantic information and emotional information through multimodal input information, providing customer service in combination with the customer's real-time emotions, and determining response plans for specific emotions. In addition to solving problems for customers, it can provide additional emotional value and improve customer experience.

[0066] Figure 1 What is shown is only a basic embodiment of the method of the present invention. By performing certain optimization and expansion on this basis, other preferred embodiments of the method can be obtained.

[0067] like Figure 2 FIG. 1 is another specific embodiment of a customer service method for real-time emotions according to the present invention. This embodiment further describes the above embodiment. In this embodiment, the method includes the following steps:

[0068] Step 201: Receive multimodal input information input by a customer; extract semantic information and emotional information from the multimodal input information.

[0069] In this embodiment, the customer service robot's sensor component / input interface (or perception layer) is used to receive multimodal information. The multimodal information includes content information and biometric information. Receiving multimodal input from the customer includes: receiving the content information via the input interface; and collecting the customer's biometric information via the sensor component.

[0070] Specifically, the sensing components in the perception layer may include: RGB-D cameras, microphone arrays, infrared thermal imagers, environmental sensors, and more. RGB-D cameras are used for facial recognition, 3D modeling, and micro-expression capture. Microphone arrays are used for sound source localization, noise reduction, and voiceprint extraction. Infrared thermal imagers are used for gait analysis, temperature monitoring, and night vision. All of the data obtained above constitutes biometric information, which refers to certain physiological characteristics displayed by the customer at the moment.

[0071] The input interface refers to the interface through which the customer service robot interacts with customers. It receives information actively input by customers, namely, content information. For example, inquiries. Content information can be text, voice, or other types of information. The input interface is the channel through which customers interact with the customer service robot.

[0072] It's understandable that the aforementioned biometric information reflects certain "states" exhibited by the customer at that moment, and this "state" can clearly reflect the customer's "emotions" to a certain extent. For example, facial micro-expressions, body movements, speech speed and tone, etc. can all reflect emotions and can be extracted from the biometric information collected by the aforementioned sensor components. In this embodiment, all types of information extracted from biometric information that represents customer emotions are collectively referred to as emotional information.

[0073] On the other hand, the content information entered by customers through the input interface will include specific inquiries, which represent their demands, or in other words, include the problems they hope to solve through customer service. In customer service, the specific meaning of this demand also needs to be analyzed.

[0074] That is, in this embodiment, the semantic information is extracted from the content information; and the emotion information is extracted from the biometric information.

[0075] Step 202: Determine query keywords based on the semantic information; and query a knowledge base based on the query keywords to determine response information.

[0076] The steps in this embodiment are similar to those of traditional customer service. That is, the query keywords are determined based on the semantic information and the specific demands are analyzed. Then, the relevant knowledge base is queried based on the query keywords to determine the response information. The response information is the answer to the "demand" or "question". In normal customer service, after receiving the response information, it can be directly fed back to the customer to answer his question. In other words, the simple and fixed-pattern question-and-answer process in the prior art can end here. However, in this embodiment, the process will be further improved and optimized through the following steps.

[0077] Step 203: Determine a real-time emotion tag based on the emotion information.

[0078] In this embodiment, in addition to providing conventional "answers" to customers, it can also further identify the customer's current emotions, that is, real-time emotions, and thus adaptively adjust the "answer" method based on emotions. In this way, in addition to answering the customer's "questions", it can also further provide a certain degree of "emotional value" to the customer, thereby improving the experience.

[0079] Combined with the aforementioned emotional information, targeted emotional analysis can be performed on the customer. Specifically, the biometric information includes facial expression information, action information, and / or voice tone information; determining a real-time emotional tag based on the emotional information includes: utilizing an analysis network to perform emotional analysis on the facial expression information, action information, and / or voice tone information to determine a corresponding emotional type; and determining the real-time emotional tag based on the emotional type.

[0080] Specifically, the biometric fusion recognition engine can be used to determine real-time emotion tags. The specific functions of the biometric fusion recognition engine are:

[0081] Face recognition, based on the ArcFace algorithm, achieved high accuracy on the LFW dataset. Voiceprint recognition: Utilizes the x-vector + PLDA model and supports cross-channel verification. Gait recognition: Analyzes spatiotemporal features through a 3D convolutional neural network (3D-CNN) to address occlusion scenarios. Micro-expression recognition: Identifies FACS action units (AUs) based on a deep residual network (ResNet-50). Speech sentiment analysis uses the Wav2Vec 2.0 pre-trained model, fine-tuned on the IEMOCAP sentiment corpus. Multimodal sentiment fusion analysis utilizes the Transformer architecture and combines a temporal alignment algorithm to synchronize speech and visual features. Furthermore, it can also provide interference-resistant biometric recognition capabilities. This involves integrating multimodal information from face, voiceprint, gait, and micro-expression to address the challenge of recognizing a single feature in the presence of occlusion (such as masks and sunglasses) or in noisy environments. For example, when a person's face is obscured, the system automatically switches to voiceprint or gait verification.

[0082] The final real-time emotion tag can indicate a single emotion of the customer, such as happiness, anger, panic, etc. It can also be a complex tag involving multiple emotions and different levels of emotion.

[0083] Step 204: Determine a corresponding emotional response plan based on the real-time emotional tag.

[0084] Combining the aforementioned response information with real-time emotional tags, a comprehensive response plan can be determined. This is the specific content that will be provided to the customer. As mentioned above, a response is a simple "answer" devoid of emotion. For example, when a user asks a simple question like, "How long will the queue last?" the response, devoid of emotion, often simply provides a time, such as "30 minutes." However, in reality, when a customer asks the same question under different emotional conditions, their mental state and desired response can be completely different. If a customer is happy, they may enjoy the excitement of the moment. In this case, the response plan can include more relevant information and interactive elements to help them fully experience the experience. Conversely, if a customer is anxious, perhaps tired of the long wait, the response plan can include guidance and comfort, or offer some entertainment to help them pass the tedious wait.

[0085] Or, for example, voice emotion and micro-expression analysis can be combined to achieve a closed loop of "emotion perception-service response". For example, when a visitor is detected to be anxious because he cannot find the store, the robot will proactively provide AR navigation and play soothing music, while delivering the mall map through the robotic arm. The service priority can also be dynamically adjusted based on the visitor's historical behavior (such as shopping frequency, consumption amount) and current emotional state. For example, a dedicated customer service takeover process is automatically triggered for VIP customers, and priority response is given to visitors with low moods, etc. In other words, it is necessary to determine a corresponding emotional response plan for real-time emotional tags.

[0086] Emotional response plans can be pre-configured in the system, for example, setting specific response plans for specific emotions. In some cases, specific response plans can also be generated for more complex emotions. In complex emotion analysis, real-time emotion tags may contain more than one emotion, each with varying degrees of intensity. In such cases, it may not be possible to accurately respond using the pre-configured emotional response plans. Instead, emotional response plans can be generated based on the actual situation.

[0087] Step 205: Determine the reply plan based on the response information and the emotional coping plan.

[0088] After determining the response information and emotional response plan, they are integrated to create a response plan. This integration can be direct, providing all relevant content to the customer in a sequential manner. Alternatively, it can mimic the normal human communication model, typically achieving both accurate communication and achieving a more ideal expression effect through direct or subtle means, thereby enhancing emotional value. This type of integration can also be achieved through artificial intelligence technology.

[0089] like Figure 3 The figure shows a specific embodiment of the customer service robot of the present invention. In this embodiment, the customer service robot is Figures 1 and 2 The execution subject of the method in the embodiment. In this embodiment, the customer service robot includes:

[0090] The sensing component 301 is used to collect biometric information.

[0091] The input interface 302 is used to receive content information.

[0092] The processor component 303 is used to extract semantic information and emotional information; determine the query keywords based on the semantic information; and query the knowledge base based on the query keywords to determine the response information; determine the real-time emotional tag based on the emotional information; and determine the reply plan based on the response information and the real-time emotional tag.

[0093] In addition, the customer service robot in this embodiment is a physical robot with an actual physical structure. Therefore, from the perspective of physical structure, it also includes: a main body structure, which includes an omnidirectional wheel chassis, a laser radar, a robotic arm, a touch screen, and a speaker.

[0094] like Figure 4 The figure shows a specific embodiment of a customer service device for real-time emotions according to the present invention. The device described in this embodiment is used to perform Figures 1-2 The physical device of the method. Its technical solution is essentially consistent with the above embodiment, and the corresponding description in the above embodiment is also applicable to this embodiment. The device in this embodiment includes:

[0095] The multimodal information receiving module 401 is configured to receive multimodal input information input by a customer and extract semantic information and emotional information from the multimodal input information.

[0096] The response information determination module 402 is configured to determine query keywords based on the semantic information, and query a knowledge base based on the query keywords to determine response information.

[0097] The emotion analysis module 403 is configured to determine a real-time emotion tag based on the emotion information.

[0098] The reply solution determination module 404 is configured to determine a reply solution based on the response information and the real-time emotion tag.

[0099] In addition Figure 4 Based on the embodiment shown, preferably, the present invention further includes:

[0100] The multimodal information receiving module 401 includes:

[0101] The content type receiving unit 411 is configured to receive the content type information via an input interface.

[0102] The biometric receiving unit 412 is used to collect the biometric information of the customer using the sensor component.

[0103] The semantic extraction unit 413 is configured to extract the semantic information from the content information.

[0104] The emotion extraction unit 414 is configured to extract the emotion information from the biometric information.

[0105] The sentiment analysis module 403 includes:

[0106] The network analysis unit 431 is configured to perform emotion analysis on the expression information, the action information and / or the tone information using an analysis network to determine a corresponding emotion type.

[0107] The label determination unit 432 is configured to determine the real-time emotion label according to the emotion type.

[0108] The response solution determination module 404 includes:

[0109] The emotion coping plan determining unit 441 is configured to determine a corresponding emotion coping plan based on the real-time emotion tag.

[0110] The reply sheet 442 is used to determine the reply plan according to the response information and the emotional coping plan.

[0111] Figure 5 : This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. Among them, the memory may include internal memory, such as high-speed random access memory (RAM), and may also include non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0112] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0113] Memory is used to store execution instructions. Specifically, execution instructions are computer programs that can be executed. Memory can include internal memory and non-volatile memory, and provides execution instructions and data to the processor.

[0114] In one possible implementation, a processor reads corresponding execution instructions from a non-volatile memory into a memory and then executes them. Alternatively, the processor may obtain corresponding execution instructions from another device to logically form a customer service device for real-time emotions. The processor executes the execution instructions stored in the memory to implement the customer service method for real-time emotions provided in any embodiment of the present invention.

[0115] The present invention Figure 4 The method performed by the medical data structured analysis device provided in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by a hardware integrated logic circuit in the processor or instructions in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The various methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0116] The steps of the method disclosed in conjunction with the embodiments of the present invention can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules within the decoding processor. The software modules can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the method described above.

[0117] The embodiment of the present invention further proposes a readable medium, which stores an execution instruction. When the stored execution instruction is executed by the processor of the electronic device, the electronic device can execute the customer service method for real-time emotions provided in any embodiment of the present invention, and is specifically used to execute the following Figure 1 or Figure 2 The method shown.

[0118] The electronic device described in each of the aforementioned embodiments may be a computer.

[0119] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or a combination of software and hardware.

[0120] The various embodiments of the present invention are described in a progressive manner. Similar portions between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so their description is relatively simple. For relevant portions, refer to the description of the method embodiments.

[0121] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0122] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A customer service method targeting real-time emotions, characterized in that: The method is applied to a customer service robot, including: Receiving multimodal input information input by a customer; extracting semantic information and emotional information from the multimodal input information; Determining query keywords based on the semantic information; and querying a knowledge base based on the query keywords to determine response information; Determining a real-time emotion tag based on the emotion information; A reply plan is determined according to the response information and the real-time emotion tag.

2. The method according to claim 1, characterized in that The multimodal information includes content information and biometric information; The receiving of multimodal input information input by the customer includes: Receiving the content information using an input interface; Use sensing components to collect customers' biometric information.

3. The method according to claim 2, characterized in that The extracting semantic information and emotional information from the multimodal input information includes: Extracting the semantic information from the content information; The emotion information is extracted from the biometric information.

4. The method according to claim 2, characterized in that The biometric information includes expression information, action information and / or tone information; and determining the real-time emotion tag based on the emotion information includes: Performing emotion analysis on the facial expression information, the action information, and / or the tone information using an analysis network to determine a corresponding emotion type; The real-time emotion tag is determined according to the emotion type.

5. The method according to any one of claims 1 to 4, characterized in that: Determining a reply plan according to the response information and the real-time emotion tag includes: Determining a corresponding emotional response plan based on the real-time emotional label; The reply plan is determined according to the response information and the emotional coping plan.

6. A customer service robot, characterized in that: include: A sensing component for collecting biometric information; Input interface, used to receive content information; A processor component for extracting semantic and emotional information; Determining query keywords based on the semantic information; and querying the knowledge base according to the query keyword to determine response information; Determining a real-time emotion tag based on the emotion information; A reply plan is determined according to the response information and the real-time emotion tag.

7. The robot according to claim 6, characterized in that: Also includes: The main body structure includes an omnidirectional wheel chassis, a laser radar, a robotic arm, a touch display screen and a speaker.

8. A customer service device for real-time emotions, characterized in that The device is placed on the customer service robot and includes: A multimodal information receiving module, configured to receive multimodal input information input by a customer; and extract semantic information and emotional information from the multimodal input information; A response information determination module is used to determine the query keyword based on the semantic information; and query the knowledge base based on the query keyword to determine the response information; A sentiment analysis module, configured to determine a real-time sentiment tag based on the sentiment information; The response plan determination module is used to determine a response plan based on the response information and the real-time emotion tag.

9. A computer-readable storage medium storing a computer program for executing the customer service method for real-time emotions according to any one of claims 1 to 7.

10. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the real-time emotional customer service method described in any one of claims 1-7.

Images