Personalized customer service system and operating method thereof

Abstract

The personalized customer service system includes client-side devices and bank-side devices. On the client-side device, an input device obtains the user's selected inquiry, a voice sensor collects the user's voice signal, a camera captures the user's facial image, and a communication device sends the inquiry, voice signal, and facial image to the bank-side device. The bank-side device's computing unit calculates the urgency level based at least on the voice signal and facial image. When the urgency level exceeds a threshold, the computing unit establishes a communication connection between the communication device and the customer service device.

Description

[Technical Field]

[0001] This invention relates to customer service systems, and more particularly to a personalized customer service system and its operation method. [Previous Technology]

[0002] Financial industry customers are increasingly demanding fast and convenient services, especially in emergency situations, where customers expect to receive support and solutions quickly.

[0003] However, when customers use existing customer service systems, they often have to go through multiple layers of system voice inquiries after making a call, followed by identity verification. This process is not only cumbersome but also time-consuming. If a customer has different services required multiple times, the same steps will be repeated, increasing the customer's time cost and resulting in a poor experience. Not to mention, in emergency situations, such as credit card fraud or ATM malfunctions, customers cannot quickly obtain the necessary support and solutions through existing customer service systems. [Summary of the Invention]

[0004] In view of the above, the present invention proposes a personalized customer service system and its operation method to solve the above problems.

[0005] An operation method of a personalized customer service system according to an embodiment of the present invention includes: an input device acquiring a consultation item selected by a user; a sound sensor collecting a sound signal of the user; a camera capturing a facial image of the user; a communication device sending the consultation item, the sound signal, and the facial image to a computing device; the computing device calculating an urgency level based at least on the sound signal and the facial image; and when the urgency level is greater than a first threshold, the computing device establishing a communication connection between the communication device and a customer service device.

[0006] A personalized customer service system according to an embodiment of the present invention includes a client device and a bank terminal device. The client device includes an input device, a sound sensor, a camera device, a first communication device, and a processor. The bank terminal device is communicatively connected to the client device. The bank terminal device includes a second communication device and a computing device. The input device is used to obtain a consultation item selected by a user. The sound sensor is used to collect a user's voice signal. The camera device is used to capture a facial image of the user. The first communication device is used to send the consultation item, the voice signal, and the facial image. The processor is electrically connected to the input device, the sound sensor, the camera device, and the first communication device. The processor is used to control the sending operation of the first communication device. The second communication device is used to receive the consultation item, the voice signal, and the facial image. The computing device is electrically connected to the second communication device. The computing device calculates an urgency level based at least on the voice signal and the facial image, and establishes a communication connection between the first communication device and a customer service device when the urgency level is greater than a first threshold.

[0007] In summary, the personalized customer service system and its operation method proposed in one embodiment of the present invention can significantly improve user experience and provide fast and efficient support. By analyzing the user's voice signals and facial expression data in real time, the system can accurately determine the urgency of the inquiry, prioritize handling emergencies in a timely manner, quickly connect to customer service personnel, reduce user waiting time, and improve response accuracy. In addition, the system can also update frequently used information services based on user behavior to ensure that users receive targeted and relevant assistance, ultimately improving user satisfaction and stickiness.

[0008] The above description of the disclosure and the following description of the embodiments are intended to demonstrate and explain the spirit and principles of the present invention, and to provide a further explanation of the scope of the patent application of the present invention.

Implementation Method

[0010] The detailed features and advantages of the present invention are described below in the embodiments. The content is sufficient to enable anyone skilled in the art to understand the technical content of the present invention and implement it accordingly. Based on the disclosure, patent claims, and drawings in this specification, anyone skilled in the art can easily understand the relevant objectives and advantages of the present invention. The following embodiments further illustrate the viewpoints of the present invention, but are not intended to limit the scope of the present invention in any way.

[0011] Figure 1 is a block architecture diagram drawn according to the first embodiment of the present invention. As shown in Figure 1, the personalized customer service system includes a client device 1 and a bank terminal device 3 that are connected to each other via a network N.

[0012] In one embodiment, network N includes one or more wide area networks (WANs), local area networks (LANs), wired networks (e.g., Ethernet), wireless networks (e.g., 802.11 networks or Wi-Fi networks), cellular networks (e.g., Long Term Evolution (LTE) networks), routers, hubs, switches, server computers, cloud computing networks, and / or combinations thereof.

[0013] As shown in FIG1, the client device 1 includes an input device 11, a sound sensor 13, a camera device 15, a first communication device 17, and a processor 19, wherein the processor 19 is electrically connected to the input device 11, the sound sensor 13, the camera device 15, and the first communication device 17. In one embodiment, the client device 1 is, for example, a laptop computer, a smartphone, a tablet computer, or an electronic device having the above-mentioned components. The present invention does not limit the hardware type of the client device 1.

[0014] As shown in FIG1, the bank terminal device 3 includes a second communication device 31 and a computing device 33, wherein the computing device 33 is electrically connected to the second communication device 31. In one embodiment, the bank terminal device 3 may be a single server, a distributed computing service cluster composed of multiple servers, or a cloud computing platform. The present invention does not limit the hardware type of the bank terminal device 3.

[0015] The input device 11 in the client device 1 is used to obtain the consultation item selected by the user. In one embodiment, the input device 11 is, for example, a keyboard, mouse, touch screen, microphone, or motion sensor. The present invention does not limit the hardware type of the input device 11. The consultation item is a service provided by the customer service system, such as reporting a lost credit card or checking a deposit account, etc., and the present invention does not limit this.

[0016] The sound sensor 13 in the client device 1 is used to collect the user's voice signal. In one embodiment, the sound sensor 13 is, for example, a microphone, a microphone array or other recording device, and the present invention does not limit the hardware type of the sound sensor 13.

[0017] The camera device 15 in the client device 1 is used to capture the user's facial image. In one embodiment, the camera device 15 includes a lens module, a CMOS sensor, an image signal processor 19 (ISP), an LED flash, a focusing motor, etc. The present invention does not limit the hardware type of the camera device 15.

[0018] The first communication device 17 in the client device 1 is used to send the consultation items selected by the user, voice signals, and facial images to the bank terminal device 3. In one embodiment, the first communication device 17 is, for example, a Wi-Fi module, a cellular data module, a Bluetooth module, an Ethernet port, an NFC (Near Field Communication) module, or a Global Positioning System (GPS) module. The present invention does not limit the hardware type of the first communication device 17.

[0019] The processor 19 in the client device 1 is used to control the transmission operation of the first communication device 17. In one embodiment, the processor 19 may be, for example, a microcontroller (MCU), an application processor (AP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system-on-a-chip (SOC), a deep learning accelerator, or any electronic device with similar functions, and the present invention is not limited thereto.

[0020] In one embodiment, the processor 19 is further configured to select one of multiple identity verification procedures based on the consultation item, and to select at least one from the voice signal or facial image for the selected identity verification procedure. In other words, the personalized customer service system will select the corresponding identity verification procedure based on the consultation item selected by the user. For example, the identity verification procedure for credit card transactions is SMS number, and the identity verification procedure for ATM deposits and withdrawals is biometric identification. Please note that the above correspondence is only an example and is not intended to limit the present invention.

[0021] In one embodiment, the processor 19 is further configured to collect multiple historical consultation items selected by the user, calculate multiple cumulative times for multiple candidate consultation items based on these historical consultation items, and adjust the sorting of the multiple candidate consultation items based on these cumulative times. Specifically, the personalized customer service system can automatically update the sorting of "frequently used customer service" items based on consultation items repeatedly used by the customer within a period of time (such as 90 days), so that the user can quickly select frequently used consultation items in the future.

[0022] In other embodiments, in addition to the above-mentioned components, the client device 1 further includes a biometric sensing device for sensing the user's physiological characteristics for identity verification. In one embodiment, the biometric sensing module is, for example, a fingerprint recognition module, a face recognition module, an iris recognition module, a voice recognition module, or a vein recognition module. The present invention does not limit the hardware type of the biometric sensing device. The face recognition module is implemented not only by using the camera device 15 (such as a front-facing camera) in the client device 1, but also includes: an infrared camera (for capturing facial images in low-light environments), a dot projector (for projecting infrared dots onto the user's face to capture three-dimensional facial features), and a flood illuminator (for providing infrared light sources in dark environments to assist in capturing facial images).

[0023] The second communication device 31 in the bank terminal device 3 is used to receive inquiries, voice signals, and facial images. The implementation of the second communication device 31 can be referred to the embodiment of the first communication device 17, and will not be repeated here.

[0024] The computing device 33 in the bank terminal device 3 is electrically connected to the second communication device 31. The computing device 33 calculates an urgency level based at least on the voice signal and facial image, and establishes a communication connection between the first communication device 17 and a customer service device when the urgency level is greater than a first threshold.

[0025] Figure 2 is a flowchart illustrating the operation method of a personalized customer service system according to an embodiment of the present invention. As shown in Figure 2, the method includes steps S1 to S7.

[0026] In step S1, the input device 11 obtains the consultation item selected by the user. In one embodiment, the bank provides a mobile banking app for users to download and run on their smartphones. The app includes the functions of the personalized customer service system proposed in this invention. After completing identity verification using an account password or biometric verification, the user can combine multiple identity verification methods (including SMS number, biometric identification, and device binding, etc.) according to business security categories or personal preferences, and then save the customized "personalized identity verification module". In addition, the user can further customize frequently used consultation items, such as credit card spending, ATM deposits and withdrawals, or loan interest calculations, and save them as "frequently used customer service services". After the above personalization settings are completed, the user can directly select "customer service hotline", and then select the consultation item through the input device 11 according to "frequently used customer service services". If the selected consultation item (such as credit card connection or deposit account inquiry) requires identity verification, the user can complete the identity verification in the mobile banking app interface according to the "personalized identity verification module".

[0027] In step S2, the sound sensor 13 collects the user's voice signal.

[0028] In step S3, the camera device 15 captures an image of the user's face.

[0029] In step S4, the processor 19 in the client device 1 controls the first communication device 17 to send the inquiry item, voice signal, and facial image to the second communication device 31 of the bank terminal device 3 via the network N, and the computing device 33 obtains the above information from the second communication device 31.

[0030] In step S5, the computing device 33 calculates the degree of urgency based at least on the sound signal and facial image. In one embodiment, the computing device 33 runs a pre-trained machine learning model to analyze and determine whether the user's physiological characteristics (such as voice speed, facial expression, pupil size, etc.) indicate an emergency state. The model can learn from a large amount of labeled data to automatically identify the characteristics of an emergency. In one embodiment, the training process of the machine learning model is as follows: First, the user's voice data and facial expression data are collected. The voice data includes voice samples from normal conversations and emergency situations. The facial expression data includes image type and video type. Then, the collected data is labeled into two categories: emergency and non-emergency. Next, features such as pitch, speech rate, and volume are extracted using sound analysis software. Features such as micro-expressions and facial muscle changes are extracted using facial recognition technology. Then, the machine learning model is trained using the labeled data. In one embodiment, the model may employ a Support Vector Machine (SVM), a Random Forest, or a Convolutional Neural Network (CNN). Cross-validation is performed during training to ensure the accuracy and reliability of the model.

[0031] In step S5, the real-time collected customer physiological data is input into the trained model, including voice speed analysis, facial expression analysis, and the extraction of features such as speech rate and micro-expressions. The model output is used to determine whether it is an emergency.

[0032] In one embodiment of step S5, the camera device 15 is further used to capture the user's facial video, and the communication device is further used to send the facial video to the computing device 33. The computing device 33 is further used to calculate the user's pupil change based on the facial video, and to increase the calculation result of the urgency level when the pupil change is greater than the second threshold.

[0033] In step S6, the computing device 33 determines whether the urgency level is greater than the first threshold. If the determination is yes, then step S7 is executed; if the determination is no, then step S2 is returned. Specifically, the model output result is used to determine whether it is an emergency. If the model outputs the probability of an emergency (i.e., the urgency level) exceeding the first threshold, then it represents an emergency situation.

[0034] In step S7, when the urgency level exceeds the first threshold, the computing device 33 establishes a communication connection between the client device and the customer service device, that is, connects the customer service device used by the customer service personnel to the first communication device 17 of the client device 1. Specifically, if it is determined to be an emergency, the customer's inquiry request is automatically set as a priority. That is, a business customer service specialist is assigned to handle the customer's reported problem immediately.

[0035] In summary, the personalized customer service system and its operation method proposed in one embodiment of the present invention can significantly improve user experience and provide fast and efficient support. By analyzing the user's voice signals and facial expression data in real time, the system can accurately determine the urgency of the inquiry, prioritize handling emergencies in a timely manner, quickly connect to customer service personnel, reduce user waiting time, and improve response accuracy. In addition, the system can also update frequently used information services based on user behavior to ensure that users receive targeted and relevant assistance, ultimately improving user satisfaction and stickiness.

[0036] Although the present invention has been disclosed above with reference to the foregoing embodiments, it is not intended to limit the present invention. Any modifications and refinements made without departing from the spirit and scope of the present invention are within the scope of patent protection of the present invention. For the scope of protection defined by the present invention, please refer to the appended claims. [Simplified Explanation of the Diagram]

[0009] Figure 1 is a block architecture diagram of a personalized customer service system according to an embodiment of the present invention; and Figure 2 is a flowchart of the operation method of a personalized customer service system according to an embodiment of the present invention.

Claims

1. A method for operating a personalized customer service system, comprising: A user selects a consultation item using an input device; The system employs the following methods: a processor selects one of multiple identity verification procedures based on the consultation item; a sound sensor collects the user's voice signal; a camera captures the user's facial image and facial video; the processor selects at least one of the voice signal or the facial image to perform the selected identity verification procedure; a communication device sends the consultation item, the voice signal, the facial image, and the facial video to a computing device; the computing device calculates the user's pupillary variability based on the facial video; the computing device calculates an urgency level based at least on the voice signal and the facial image, where the urgency level is the probability of an emergency, and when the pupillary variability exceeds a second threshold, the computing device increases the calculated urgency level; and when the urgency level exceeds a first threshold, the computing device establishes a communication connection between the communication device and a customer service device.

2. The method of operating the personalized customer service system as described in claim 1 further includes: A single processor collects multiple historical consultation items selected by the user. The processor calculates multiple cumulative counts for multiple candidate consulting projects based on these historical consulting projects; and the processor adjusts the ranking of these candidate consulting projects based on these cumulative counts.

3. A personalized customer service system, comprising: A client device includes: an input device for acquiring a consultation item selected by a user; a sound sensor for collecting a sound signal from the user; a camera device for capturing a facial image and a facial video of the user; a first communication device for transmitting the consultation item, the sound signal, the facial image, and the facial video; a processor electrically connected to the input device, the sound sensor, the camera device, and the first communication device, the processor controlling the transmission operation of the first communication device; the processor is further configured to select one of multiple identity verification procedures based on the consultation item, and to select at least one from the sound signal or the facial image to perform the selected identity verification procedure; and a bank terminal device communicatively connected to the client device includes: a second communication device for receiving the consultation item, the sound signal, and the facial image. A computing device electrically connected to the second communication device, the computing device at least calculates an urgency level based on the sound signal and the facial image, and calculates the pupil change of the user based on the facial image, wherein the urgency level is the probability of an emergency, and when the pupil change is greater than a second threshold, the computing device increases the calculation result of the urgency level, and when the urgency level is greater than a first threshold, establishes a communication connection between the first communication device and a customer service device.

4. The personalized customer service system as described in claim 3, wherein the processor is further configured to collect multiple historical consultation items selected by the user, calculate multiple cumulative counts of multiple candidate consultation items based on the historical consultation items, and adjust the order of the candidate consultation items based on the cumulative counts.

Images