Digital human interaction method and system based on multi-modal biological feature recognition

Abstract

The invention discloses a digital human interaction method and system based on multi-modal biological feature recognition. The method comprises the steps that a target reception signal is received, a target reception verbal skill in a knowledge base is acquired based on the target reception signal, and the target reception signal is a signal for receiving a target customer; when it is recognized that the target customer arrives at a target position, target scene interaction corresponding to the target position is started based on the target reception verbal skill; obtaining an interaction state of the target customer in real time, and performing service quality analysis based on the interaction state to obtain a target service quality analysis result; and optimizing the target reception verbal skill based on the target service quality analysis result, and putting the target service quality analysis result into a service quality evaluation library so as to optimize the knowledge base based on the service quality evaluation library. According to the digital human interaction method based on multi-modal biological feature recognition provided by the invention, the human-computer interaction experience when the user uses the digital human customer service is improved.

Description

Technical Field

[0001] The present invention relates to the field of digital human-computer interaction, and in particular to a digital human interaction method and system based on multimodal biometric feature recognition. Background Art

[0002] With the continuous development of science and technology, a new generation of customer service systems represented by digital humans are gradually emerging in areas such as smart exhibition halls, smart front desks and voice intelligent customer service, and are expected to gradually replace traditional manual services.

[0003] At present, digital human customer service is mainly based on text-based answering, which matches the input text with the question and answer content in the knowledge base and outputs the corresponding answer in text form. More advanced is voice intelligent customer service, which can extract the user's questions and intentions through voice recognition and semantic understanding, and then find the most matching answer from the relevant scope and reply through voice.

[0004] The latest digital human customer service adds outputs such as virtual human animation movements, gestures, tone and expressions on the basis of voice customer service to simulate the interactive behaviors between people. This customer service system is mainly used in scenarios that require real-person reception, such as exhibition halls.

[0005] However, both the popular voice intelligent customer service and the new generation of digital human intelligent customer service have a common problem, that is, the phenomenon of irrelevant answers. This situation is likely due to the intelligent customer service's error in recognizing the user's voice, resulting in deviations in semantic understanding or intention understanding, and ultimately making the user's answer inconsistent with the question. Therefore, how to improve the service quality and accuracy of intelligent customer service has become an urgent problem to be solved.

[0006] Therefore, the prior art still needs to be improved and enhanced. Summary of the invention

[0007] The present invention provides a digital human interaction method and system based on multimodal biometric recognition, aiming to solve the problem of low user satisfaction caused by voice recognition errors and semantic understanding deviations in the service scenarios of digital human customer service in the prior art.

[0008] A first aspect of the present invention provides a digital human interaction method based on multimodal biometric recognition, which is used to interact with a target customer, comprising:

[0009] receiving a target reception signal, and acquiring a target reception speech in a knowledge base based on the target reception signal, wherein the target reception signal is a signal for receiving a target customer;

[0010] When it is identified that the target customer has arrived at the target location, a target scene interaction corresponding to the target location is initiated based on the target reception script;

[0011] Acquire the target customer's interaction status in real time, and perform service quality analysis based on the interaction status to obtain a target service quality analysis result;

[0012] The target reception speech is optimized based on the target service quality analysis result and the target service quality analysis result is placed in a service quality evaluation library, so as to optimize the knowledge base based on the service quality evaluation library. In one implementation, the target reception speech in the knowledge base is obtained based on the target reception signal, including:

[0013] Acquire target identity information of the target customer based on the target reception signal, wherein the target identity information includes the age and job type of the target customer;

[0014] The reception speech matching the target identity information in the knowledge base is obtained as the target reception speech.

[0015] In one implementation, starting the target scene interaction corresponding to the target location based on the target reception speech includes:

[0016] Retrieving target interactive content in a knowledge base based on the target position;

[0017] Explain and ask questions about the product corresponding to the target location based on the target interactive content;

[0018] The target interactive content is dynamically displayed on a display screen at the target location in real time.

[0019] In one implementation, the real-time acquisition of the target customer's interaction status includes:

[0020] Perform voiceprint recognition, speech recognition, emotion recognition and auxiliary recognition on the target customer to collect the target interaction data of the target customer, wherein the target interaction data includes the voiceprint emotion, speech content, emotion expression conclusion and body expression conclusion corresponding to each interaction stage;

[0021] The interaction status of the target customer is obtained based on the target interaction data.

[0022] In one implementation, performing service quality analysis based on the interaction status includes:

[0023] Identifying the proportion of positive emotion expressions corresponding to the target interaction stage based on the target interaction data corresponding to the target interaction stage, and if the proportion of positive emotion expressions is greater than a first preset value, determining that the service quality corresponding to the target interaction stage is excellent;

[0024] If the proportion of positive emotion expression is greater than a second preset value and less than or equal to the first preset value, then the service quality corresponding to the target interaction stage is determined to be qualified;

[0025] If the proportion of positive emotion expression is greater than the third preset value and less than or equal to the second preset value, it is determined that the service quality corresponding to the target interaction stage is average;

[0026] If the proportion of positive emotion expressions is less than or equal to the third preset value, it is determined that the service quality corresponding to the target interaction stage is to be optimized.

[0027] In one implementation, the optimizing the target reception speech based on the target service quality analysis result includes:

[0028] Adjusting the target reception speech based on the target service quality analysis result until the service quality of the service quality analysis result is qualified or excellent, and recording the adjusted target reception speech;

[0029] Matching the target customer's voiceprint information with the adjusted target reception speech, and recording the information in the knowledge base, so that the adjusted target reception speech is automatically matched when providing other customer service to the target customer;

[0030] The target customer is then received based on the target reception script.

[0031] In one implementation, the optimizing the knowledge base based on the service quality evaluation base includes:

[0032] Acquire a service improvement experience database based on the service quality analysis results corresponding to different identity information in the service quality evaluation database;

[0033] Based on the service improvement experience database, the explanation speech bias corresponding to different identity information in the knowledge base is optimized and updated.

[0034] A second aspect of the present invention is a digital human interaction system based on multimodal biometric recognition, comprising:

[0035] A signal receiving module receives a target reception signal, and acquires a target reception speech in a knowledge base based on the target reception signal, wherein the target reception signal is a signal for receiving a target customer;

[0036] An interactive module, for initiating a target scene interaction corresponding to the target location based on the target reception script when recognizing that the target customer has arrived at the target location;

[0037] An analysis module, used to obtain the interaction status of the target customer in real time, and perform service quality analysis based on the interaction status to obtain a target service quality analysis result;

[0038] The optimization module is used to optimize the target reception speech based on the target service quality analysis result and put the target service quality analysis result into a service quality evaluation library, so as to optimize the knowledge base based on the service quality evaluation library.

[0039] According to a third aspect of the present invention, a terminal is provided, wherein the terminal comprises: a memory, a processor, and a digital human interaction program based on multimodal biometric recognition stored in the memory and executable on the processor, wherein the digital human interaction program based on multimodal biometric recognition, when executed by the processor, implements the steps of the digital human interaction method based on multimodal biometric recognition as described above.

[0040] According to a fourth aspect of the present invention, a storage medium is provided, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any of the above-mentioned digital human interaction methods based on multimodal biometric recognition.

[0041] Beneficial effects: Compared with the prior art, the present invention provides a digital human interaction method and system based on multimodal biometric identification. In the digital human interaction method based on multimodal biometric identification provided by the present invention, by receiving a target reception signal, the target reception speech in the knowledge base is obtained based on the target reception signal, wherein the target reception signal is a signal for receiving the target customer. Then, when it is recognized that the target customer has arrived at the target location, the target scene interaction corresponding to the target location is started based on the target reception speech. During the interaction process, the interaction status of the target customer is obtained in real time, and a service quality analysis is performed based on the interaction status to obtain a target service quality analysis result. Finally, the target reception speech is optimized based on the target service quality analysis result and the target service quality analysis result is placed in a service quality evaluation library to optimize the knowledge base based on the service quality evaluation library. The present invention provides users with a digital human interaction method based on multimodal biometric recognition, which solves the problem of low user satisfaction caused by voice recognition errors and semantic understanding deviations in the service scenarios of digital human customer service in the prior art. While improving the human-computer interaction experience of digital human intelligent customer service, it also improves the service quality of digital human customer service, provides users with a more comfortable digital human customer service interaction method, and improves the exhibition experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1A flowchart of an embodiment of a digital human interaction method based on multimodal biometric recognition provided by the present invention;

[0043] Figure 2 A schematic diagram of a scenario of an embodiment of a digital human interaction method based on multimodal biometric recognition provided by the present invention;

[0044] Figure 3 A flow chart of the customer service of a digital human in an exhibition hall in an embodiment of the digital human interaction method based on multimodal biometric recognition provided by the present invention;

[0045] Figure 4 A structural principle diagram of an embodiment of a digital human interaction system based on multimodal biometric recognition provided by the present invention;

[0046] Figure 5 A schematic diagram of an operating environment of an embodiment of a terminal provided by the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0048] Embodiment 1

[0049] The digital human interaction method based on multimodal biometric recognition provided in this embodiment can be executed by a terminal, which can be, but is not limited to, a computer, a smart mobile device, etc., and is described below using a computer as an example. In this embodiment, the terminal is a server equipped with a digital human customer service system, which is mainly used to obtain the interactive status of the digital human customer service and the service object, and optimize and improve the digital human customer service system based on the interactive status.

[0050] In the existing technology, intelligent customer service systems, whether they are popular voice intelligent customer service or the new generation of digital human intelligent customer service, often encounter a problem: the answer does not match the user's question. This is usually because the system makes mistakes in the voice recognition link, resulting in misunderstanding of the user's semantics or misjudgment of the user's intention. Therefore, users often feel that the answers of intelligent customer service are irrelevant.

[0051] In order to improve and enhance the service quality of intelligent customer service and the relevance and accuracy of service content, especially for scenarios such as digital human exhibition hall reception, which have extremely high requirements for service quality and response accuracy - because this is directly related to the display of corporate image and service capabilities, in this embodiment, an effective method is provided to collect reception information data, and to evaluate and improve the performance of intelligent customer service based on this.

[0052] Specifically, Figure 1 As shown, the digital human interaction method based on multimodal biometric recognition provided in this embodiment includes the following steps:

[0053] S100, receiving a target reception signal, and acquiring a target reception speech in a knowledge base based on the target reception signal, wherein the target reception signal is a signal for receiving a target customer.

[0054] Digital human customer service is mainly used to receive corresponding customers. When a customer enters the range of digital human customer service, a reception signal will be sent to the digital human customer service automatically or by manual input. In this embodiment, when the target customer enters the service range of digital human customer service, the target reception signal will be sent to the digital human customer service. The target reception signal includes the target identity information of the target customer and the identification data of the target customer.

[0055] In this embodiment, a digital human interaction method based on multimodal biometric recognition provided in this embodiment is implemented in a smart exhibition hall as an example for detailed description.

[0056] Specifically, in a smart exhibition hall, when the digital human interaction method based on multimodal biometric recognition provided by this embodiment is implemented, before obtaining the target reception information, it also includes:

[0057] Construct a digital human customer service system, which includes a built-in digital human, an external digital human, and a digital human robot. The built-in digital human is set inside the smart device in the exhibition hall, and the external digital human is set outside the exhibition equipment in the exhibition hall to serve the corresponding exhibition equipment. The digital human robot is used to receive customers visiting the exhibition;

[0058] The digital human customer service system is connected to the smart exhibition hall intelligent system to obtain the visit trajectory of the received customers and assist in obtaining the audio and video elements of the sound, expression, and action of the customer reception scene. The smart exhibition hall intelligent system includes millimeter wave sensors, cameras, and microphone pickup arrays.

[0059] Specifically, in this embodiment, a digital human interaction method supported by advanced multimodal biometric recognition technology is proposed. Taking the smart exhibition hall as an example, the digital human customer service system is first constructed in the smart exhibition hall, and the digital human customer service system consists of several key modules: digital human interaction module, knowledge base module, reception management module, service quality analysis module and equipment management module.

[0060] Among them, the digital human interaction module allows users to customize the appearance and behavioral characteristics of the digital human; the knowledge base module is responsible for maintaining the interactive content and ensuring that the digital human can provide appropriate information to respond to customer inquiries; the reception management module is specifically used for exhibition hall reception, which can identify customers and record their interactions with digital humans and on-site reception staff.

[0061] The service quality analysis module is the core of the digital human customer service system, which can activate or deactivate multiple recognition modules, including voice recognition module, voiceprint recognition module, action recognition module and emotion recognition module. The voiceprint recognition module is used to identify and recognize returning customers; the emotion recognition module is used to evaluate the customer's reaction and attention to the digital human's reply. Furthermore, the emotion recognition module will analyze the customer's emotional state, such as approval, doubt or denial, based on multi-dimensional data such as voice, body language and expression, so as to adjust the digital human's question and answer library in real time.

[0062] All these analyses help measure the effectiveness of interactions between customers and digital humans and optimize interactive content accordingly. The analysis results will be saved in the service quality evaluation library and service improvement experience library for subsequent use. Finally, the service quality improvement module will refer to these data for review, whether automatically or manually, to ensure that the content of the knowledge base is continuously updated and improved to continuously improve the service experience.

[0063] Specifically, when the digital human customer service system and the smart exhibition hall intelligent system are built, the smart exhibition hall is considered ready, wherein the smart exhibition hall intelligent system includes millimeter wave sensors, cameras, microphone pickup arrays, etc.

[0064] When the smart exhibition hall is ready, the intelligent sensing equipment in the exhibition hall is started, including millimeter waves, surveillance cameras, etc. At the same time, the exhibition hall exhibits will also start, such as the digital customer service robot and display products such as TVs, washing machines, and display screens.

[0065] Then, the digital human customer service system is started. The digital human customer service system includes a built-in digital human, an external digital human and a digital human robot. The built-in digital human is set inside the smart device of the exhibition, and the external digital human is set near the non-smart device of the exhibition. The digital human robot is used to receive customers visiting the exhibition. The digital human customer service system has a digital human interaction module, a knowledge base module, a reception management module, a service quality analysis module and an equipment management module. For details, refer to Figure 2 .

[0066] After the digital human customer service system is started, it first establishes a connection with the smart exhibition hall intelligent system. The smart exhibition hall intelligent system includes millimeter wave sensors, cameras, microphone pickup arrays, etc., which are used to obtain the visit trajectory of the reception customers and assist in obtaining the audio and video elements of the sound, expression, and action of the reception customer scene.

[0067] In this embodiment, all digital customer service robots in the smart exhibition hall have their own private systems; the TV and system in the smart exhibition hall are also equipped with built-in virtual digital human systems; the external display devices in the smart exhibition hall are also equipped with virtual digital human systems, wherein all digital customer service robots have their own digital human systems, TV and system built-in virtual digital human systems, and external display devices build virtual digital human systems. They are all connected to the server of the digital human customer service system through the network. The digital human interaction module in the digital human customer service system manages and distributes interactive images for all digital humans or display devices loaded with virtual digital humans in the exhibition hall; the knowledge base module in the digital human customer service system is mainly used for knowledge base management and information distribution; the reception management module in the digital human customer service system is mainly used for reception management and reception information distribution; the service quality analysis module in the digital human customer service system is mainly used for customer reception data recording and service quality data collection results analysis and service quality improvement strategy distribution and service quality scoring; the device management module in the digital human customer service system manages the device terminals of distributed digital humans.

[0068] Specifically, during the connection between the digital human customer service system and the smart exhibition hall intelligent system, the service digital human of the exhibition hall equipment of the smart exhibition hall will connect to the digital human customer service system server after the equipment is turned on. Specifically, the digital human equipment / system establishes a connection mechanism with the exhibition hall equipment, obtains the online status of the equipment and the digital human and the mutual connection status, and reports the equipment status, which includes online or offline, digital human number, such as "SZR-ZT01-202300001AA", digital human service equipment, such as "TV-ZT01-202300001AA" and reception object and other information. The digital human robot obtains the reception arrangement from the reception management module. If there is no reception arrangement, the field corresponding to the digital human service equipment is "none". If there is a reception, the field is the number of the reception arrangement, such as "KK-202305130001A".

[0069] Furthermore, if the digital human providing services for the exhibition hall equipment in the smart exhibition hall fails to connect to the server of the digital human customer service system after the equipment is turned on, the digital human will inform the staff of the online status of the digital human who cannot connect to the server of the digital human customer service system.

[0070] For the external digital human, if it can connect to the server of the digital human customer service system but cannot connect to the corresponding exhibition hall equipment and there is only an external digital human, an error will be reported to the device management module of the digital human customer service system server, and the online status of the corresponding device of the external digital human machine will be informed to the staff, such as "Device A is not online, please check the status of device A, digital human A cannot connect to the device."

[0071] If the status of a digital human in the equipment list cannot be obtained after the digital human customer service system server is turned on, the staff will be notified of the digital human's online status, such as "Failed to obtain the status of the digital human of device B". After receiving the notification, the staff can manage the digital human and the service exhibition hall equipment through the device management module.

[0072] After all the work is ready, the smart exhibition hall can start receiving users.

[0073] Reference Figure 3 When a customer enters the exhibition hall, the staff of the exhibition hall will input the target reception information into the management terminal of the digital human customer service system. The target reception information includes the identity identification information of the target customer. After receiving the target reception information, the digital human customer service system will send it to all the digital humans in the exhibition hall through the server of the digital human customer service system, and assign the target digital human as the reception customer service of the target customer based on the target reception information.

[0074] Specifically, the step of acquiring the target reception speech in the knowledge base based on the target reception signal includes:

[0075] Acquire target identity information of the target customer based on the target reception signal, wherein the target identity information includes the age and job type of the target customer;

[0076] The reception speech matching the target identity information in the knowledge base is obtained as the target reception speech.

[0077] Specifically, since the target reception signal includes the target identity information of the target customer and the identification data of the target customer, the target reception words corresponding to the target customer can be extracted from the knowledge base based on the target identity information in the target reception signal, and based on the identification data of the target customer in the target reception signal, the location of the target customer can be identified in real time.

[0078] S200: When it is identified that the target customer has arrived at the target location, the target scene interaction corresponding to the target location is started based on the target reception script.

[0079] In this embodiment, the target customer can be a customer in a large exhibition hall, a large shopping mall, or any place that needs to be received, and the digital human customer service is connected to the smart devices in the place, and can obtain the real-time location of the target customer based on the intelligent millimeter wave sensor, camera or microphone pickup array. Therefore, when it is identified that the target customer has arrived at the place where the digital human customer service service is needed, in this embodiment, it is defined as the target location. When it is identified that the target customer has arrived at the target location, the target scene interaction corresponding to the target location will be started based on the target reception script.

[0080] Specifically, starting the target scene interaction corresponding to the target location based on the target reception speech includes:

[0081] S210, retrieving target interactive content in a knowledge base based on the target position;

[0082] S220, explaining the product corresponding to the target location and asking questions based on the target interactive content;

[0083] S230: Dynamically display the target interactive content on a display screen at the target location in real time.

[0084] Specifically, taking the smart exhibition hall as an example, the intelligent sensing device of the smart exhibition hall intelligent system will obtain the target customer's visit trajectory in real time, and notify the digital human customer service corresponding to the booth that is about to arrive to prepare for reception. At the same time, the intelligent sensing devices near the target customer's visit trajectory, such as cameras and microphone pickup arrays, are mobilized. The intelligent sensing device is used to follow the trajectory of the target customer to collect audio and video data in real time, such as obtaining the dialogue exchanges generated during the user's visit, and send the collected information to the digital human customer service system in real time for the digital human customer service system to perform real-time quality analysis, and at the same time, enrich the knowledge of the knowledge base in the digital human customer service system.

[0085] The intelligent sensing device of the intelligent system of the smart exhibition hall continuously obtains the visit trajectory of the received customers and keeps the visit trajectory synchronized with the digital human customer service system. Specifically, the intelligent system of the smart exhibition hall makes a joint judgment through the judgment of the camera of the intelligent sensing device and the camera of the digital human device. The camera of the intelligent sensing device judges the area where the visit trajectory is located. If the target customer arrives at the target booth, the target arrival information will be sent to the digital human customer service system. After receiving the target arrival notification, the digital human customer service system will also confirm whether the target customer is visiting the display equipment of the target booth through the camera of the digital human device in the target booth. If it is recognized that the target customer has arrived at the display location of the display equipment in the target booth, the digital human customer service system will call the corresponding knowledge base speech and resources to interact with the customer according to the reception arrangement of the target reception information.

[0086] Specifically, the digital human customer service system will retrieve the target interactive content from the knowledge base based on the target arrival notification and send it to all the digital human customer service staff at the target booth. All the digital human customer service staff at the target booth will explain the exhibits at the target booth and ask questions based on the target interactive content.

[0087] The interaction between the digital human customer service and the target customer will be dynamically displayed in real time according to the interaction status. Specifically, the target interactive content will be dynamically displayed on the display screen of the target booth in real time. The display screen can be a display screen of a built-in digital human, or a display screen of an external digital human corresponding to the exhibition equipment, or a display screen of a digital human robot.

[0088] S300: Acquire the interaction status of the target customer in real time, and perform service quality analysis based on the interaction status to obtain a target service quality analysis result.

[0089] The real-time acquisition of the target customer's interaction status includes:

[0090] Perform voiceprint recognition, speech recognition, emotion recognition and auxiliary recognition on the target customer to collect the target interaction data of the target customer, wherein the target interaction data includes the voiceprint emotion, speech content, emotion expression conclusion and body expression conclusion corresponding to each interaction stage;

[0091] The interaction status of the target customer is obtained based on the target interaction data.

[0092] That is to say, by identifying the target customer's voice, tone, emotion and body expression, the target customer's interaction status can be identified. The interaction status is the satisfaction recognition result of the target customer when interacting in the customer service scenario. Based on the satisfaction recognition result, the current service quality can be analyzed to obtain real-time analysis results of the target service quality.

[0093] Specifically, taking the smart exhibition hall as an example, during the interaction between the digital human customer service and the target customer, and the interaction between the personnel participating in the reception of the target customer and the reception customer, the device carrying the digital human customer service and the intelligent sensing device will collect the target customer's interactive data in real time and record the reception data (reception data record. Specifically, during this visit, the digital human customer service identifies the voiceprint of the person participating in the reception scene and matches the voiceprint through the face recognition of the target customer, and records the interactive content corresponding to the voiceprint feature, so as to facilitate the subsequent sorting of the reception content and the targeted tuning of the reception knowledge base. When the voiceprint feature of the target customer is extracted for the first time, it will be reported to the service quality analysis module of the digital human customer service system server. The service quality analysis module establishes a reception content database with the reception number as the serial number. The reception content corresponding to each target customer includes: voiceprint ID, information collection digital human ID / information collection intelligent sensing device ID, voiceprint collection content, etc. At the same time, other devices in the smart exhibition hall will also report the collected voiceprint and the content corresponding to the voiceprint to the database through the digital human during this reception process for unified management.

[0094] Specifically, in this embodiment, multimodal biometric recognition includes but is not limited to "voiceprint recognition", "speech recognition", "emotion recognition" and "auxiliary recognition" to obtain the interaction status between the target customer and the digital human customer, and conduct service quality analysis.

[0095] Specifically, the voiceprint recognition is used to distinguish who is speaking, and record the content of the speech and the multimodal biometric recognition result data and collect them into the reception content database of the digital human customer service system server;

[0096] The speech recognition is used to record the conversation content said during the reception process, including the sound content displayed by the digital human, which is collected into the reception content database of the digital human customer service system server according to the voiceprint features;

[0097] The emotion recognition is used to collect the emotional expressions of visitors when the digital person explains a certain content through sound collection and image collection, and judge the reception degree of the visitors to the explanation content (satisfied, average, dissatisfied, uninterested, interested, etc.) through emotions, including the emotional expressions of visitors during the conversation with the receptionist;

[0098] Other features for auxiliary identification include body movement recognition such as nodding, shaking head, etc., which are used to assist in identifying the customer's emotions and status.

[0099] Based on the multimodal biometric feature recognition, the real-time interaction status of the target user is obtained, so as to perform service quality analysis based on the interaction status and understand the satisfaction of the target user with the electronic customer service system.

[0100] The performing of service quality analysis based on the interaction status includes:

[0101] Identifying the proportion of positive emotion expressions corresponding to the target interaction stage based on the target interaction data corresponding to the target interaction stage, and if the proportion of positive emotion expressions is greater than a first preset value, determining that the service quality corresponding to the target interaction stage is excellent;

[0102] If the proportion of positive emotion expression is greater than a second preset value and less than or equal to the first preset value, then the service quality corresponding to the target interaction stage is determined to be qualified;

[0103] If the proportion of positive emotion expression is greater than the third preset value and less than or equal to the second preset value, it is determined that the service quality corresponding to the target interaction stage is average;

[0104] If the proportion of positive emotion expressions is less than or equal to the third preset value, it is determined that the service quality corresponding to the target interaction stage is to be optimized.

[0105] Specifically, the server of the digital human customer service system stores a special service quality evaluation library, which stores quality analysis records created based on user interactions. These records include in detail the digital human ID, reception ID, explanation content ID, and the results of multimodal biometric feature recognition data collected from the target user.

[0106] For example, the biometric data collection results for a specific target user may show that the user with voiceprint IDXX is satisfied with a certain content and has made a specific reaction and emotional expression. The conclusion of the emotional expression may be a positive or negative emotional reaction.

[0107] For a specific explanation content, during the reception process, if the monitored positive emotional expressions obviously exceed the negative emotional expressions, a more in-depth analysis can be performed. Specifically, if the proportion of positive emotions exceeds the first preset value, the service quality is rated as excellent; if the proportion of positive emotions is between the second preset value and the first preset value, it is considered qualified; and if the proportion of positive emotions is between the third preset value and the second preset value, the service quality is considered average; if it is lower than the third preset value, it is considered to need optimization. In this embodiment, the first preset value is 95%, the second preset value is 80%, and the third preset value is 65%. Such an evaluation system helps to systematically analyze and improve the overall quality of the digital human customer service system.

[0108] S400: Optimize the target reception speech based on the target service quality analysis result and put the target service quality analysis result into a service quality evaluation library, so as to optimize the knowledge base based on the service quality evaluation library.

[0109] The optimizing the target reception speech based on the target service quality analysis result includes:

[0110] Adjusting the target reception speech based on the target service quality analysis result until the service quality of the service quality analysis result is qualified or excellent, and recording the adjusted target reception speech;

[0111] Matching the target customer's voiceprint information with the adjusted target reception speech, and recording the information in the knowledge base, so that the adjusted target reception speech is automatically matched when providing other customer service to the target customer;

[0112] The next reception of the target customer is carried out based on the adjusted target reception words.

[0113] Specifically, the digital human customer service system that provides services at the same place can dynamically adjust the corresponding customer reception words according to the service quality analysis results during the reception process. That is to say, when the digital human customer service system serves the target customer, it can perform service quality analysis according to each interaction status of the target customer to obtain the corresponding target service quality analysis results, and then adjust and optimize the target customer's next reception words based on the target service quality analysis results until the words that best suit the target customer are found, and the next reception service is performed as the target reception words. Specifically, for users with the same voiceprint ID, such as the words corresponding to the voiceprint ID of the target user, when the next scene interaction corresponding to the next location is performed, the knowledge base words that are more in line with the emotional characteristics of the target customer are retrieved as the adjusted target reception words for the next exhibition customer service. That is to say, the digital human customer service will obtain the interest range of the customer identified in the current reception in the knowledge base according to the content of interest to the target user, and adjust the reception words and explanation content to meet the interests of the target customer. For example, if the digital human customer service system detects that a user with a certain voiceprint ID has a good emotional response to a specific speech, then when the user enters the next service location, the digital human will automatically select the speech in the knowledge base that matches the user's emotional characteristics to continue to provide customer service. This personalized service method can ensure that the customer receives a consistent and personalized customer service experience throughout the service process.

[0114] In addition, the digital human customer service can also dynamically adjust the depth and style of its explanations based on the specific attributes of the visitors, such as their work backgrounds, whether they are technicians, sales staff or senior managers. The system will customize the bias of the explanation content according to the background and needs of different customers, so as to better fit the professional interests and expectations of each customer. This series of customized interactions is designed to improve the overall customer satisfaction and enhance their understanding of the displayed content. Based on this, after data collection and modeling analysis of multimodal biometric recognition, matching the attributes of the visiting customers, the reception words of the digital human customers can be adjusted in real time by dynamically collecting the service quality analysis results.

[0115] In this way, the digital human customer service system is not just an interface that provides information, but has become a highly intelligent reception assistant that continuously improves service strategies by deeply analyzing customer feedback to ensure that every customer can get the highest quality reception experience. This dynamic adjustment mechanism reflects the flexibility and adaptability of the digital human customer service system, enabling it to maintain outstanding performance in a constantly changing service environment.

[0116] After optimizing the target reception speech corresponding to the target customer based on the target service quality analysis result, the method further includes placing the target service quality analysis result into a service quality evaluation library to optimize the knowledge base based on the service quality evaluation library.

[0117] The optimizing the knowledge base based on the service quality evaluation base includes:

[0118] Acquire a service improvement experience database based on the service quality analysis results corresponding to different identity information in the service quality evaluation database;

[0119] Based on the service improvement experience database, the explanation speech bias corresponding to different identity information in the knowledge base is optimized and updated.

[0120] In this embodiment, after optimizing the target reception speech corresponding to the target customer, the target service quality analysis result will also be put into the service quality evaluation library. The service quality evaluation library is automatically generated by the service quality analysis record corresponding to the explanation speech retrieved by the knowledge base module. It records the details and quality analysis results of each service, and provides an evaluation benchmark and improvement direction for subsequent services.

[0121] Then, a service improvement experience library is obtained based on the service quality analysis results corresponding to different identity information in the service quality evaluation library. In this embodiment, the service improvement experience library is maintained manually based on the optimization information obtained from the service quality evaluation library and other channels, and needs to be formed after in-depth analysis based on the service quality evaluation and the attributes of the visiting customers. This library takes into account the attributes of each visiting customer, such as technicians, sales personnel, R&D managers, sales managers, group leaders, etc., because the explanation language tends to be different for each customer attribute, and it needs to be easy to fit the background and needs of the visiting customers. In this way, the service improvement experience library becomes a valuable resource for guiding how to customize the explanation content and style according to different customer groups.

[0122] At the same time, the maintenance and update of the knowledge base of the digital human customer service system is also an important link. The content in the service improvement experience library will be regularly updated to the knowledge base to ensure that the knowledge base always maintains relatively cutting-edge service strategies and information. The knowledge base contains attribute dimensions corresponding to different types of customers, which enables the explanation of the same interactive content to be carried out along different explanation directions to adapt to the interaction of users with different backgrounds.

[0123] For example, the focus of explanations will be very different for technicians and salespeople. Technicians may pay more attention to the details of product technology implementation, including technical principles, implementation plans, performance indicators, etc., while salespeople may be more concerned with business-related information such as product prices, market positioning, service support, etc. Therefore, by adjusting the explanation language to match the needs and interests of different customers, the accuracy and effectiveness of the interaction can be greatly improved.

[0124] In summary, the digital human customer service system continuously improves the professionalism of services and customer satisfaction through dynamic service optimization, detailed quality analysis, and personalized speech customization, thereby achieving efficient and targeted information transmission and customer interaction in the practice of receiving different customers. This intelligent service model not only improves efficiency, but also brings higher service levels and better brand image to enterprises.

[0125] Specifically, this embodiment uses voiceprint recognition, action recognition, emotion recognition, and speech recognition of the service quality analysis module. Voiceprint recognition is used to mark and identify customers, and emotion recognition is used to analyze the acceptance and attention of the digital human's reply to customers. Through the emotion recognition (voice, body, expression, etc.) generated by the interaction between the customer and the digital human and the receptionist (multi-dimensional analysis such as recognition, doubt, and disapproval), the content of the digital human question and answer library and the matching degree between the receptionist's dialogue and the user's perception are dynamically obtained. The interactive content of the digital human question and answer library is dynamically optimized according to the perceived matching degree, and the analysis results are output to the service quality evaluation library and the service improvement experience library; the service quality improvement module performs automatic or manual review according to the marking of the service quality experience library, updates the content to the knowledge base, and improves the content system of the knowledge base. In order to improve the digital human customer service system, a multimodal and scenario-based service quality data collection, analysis, and optimization method based on the distributed digital human system is provided, which improves the human-computer interaction experience of the digital human intelligent customer service while improving the service quality of the digital human customer service.

[0126] In summary, this embodiment provides a digital human interaction method based on multimodal biometric recognition, by receiving a target reception signal, and obtaining a target reception speech in a knowledge base based on the target reception signal, wherein the target reception signal is a signal for receiving a target customer. Then, when it is recognized that the target customer has arrived at the target location, the target scene interaction corresponding to the target location is started based on the target reception speech. During the interaction process, the interaction status of the target customer is obtained in real time, and a service quality analysis is performed based on the interaction status to obtain a target service quality analysis result. Finally, the target reception speech is optimized based on the target service quality analysis result and the target service quality analysis result is placed in a service quality evaluation library to optimize the knowledge base based on the service quality evaluation library. This embodiment provides a digital human interaction method based on multimodal biometric recognition for users, which solves the problem of low user satisfaction caused by voice recognition errors and semantic understanding deviations in the service scenarios of digital human customer service in the prior art, and improves the human-computer interaction experience of digital human intelligent customer service while improving the service quality of digital human customer service, bringing users a more comfortable digital human customer service interaction method and improving the exhibition experience.

[0127] It should be understood that, although the various steps in the flowcharts given in the accompanying drawings of the present invention are displayed in sequence according to the indications of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, the execution of the steps in the present invention is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps of the present invention may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0128] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by signaling related hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0129] Embodiment 2

[0130] Based on the above embodiments, the present invention also provides a digital human interaction system based on multimodal biometric recognition, and its functional module diagram is shown in FIG. Figure 4 As shown, the digital human interaction system based on multimodal biometric recognition includes, as specifically described in the first embodiment:

[0131] A signal receiving module receives a target reception signal, and acquires a target reception speech in a knowledge base based on the target reception signal, wherein the target reception signal is a signal for receiving a target customer, as described in the first embodiment;

[0132] An interactive module, for initiating a target scene interaction corresponding to the target location based on the target reception script when recognizing that the target customer has arrived at the target location, as specifically described in the first embodiment;

[0133] An analysis module, used to obtain the interaction status of the target customer in real time, and perform service quality analysis based on the interaction status to obtain a target service quality analysis result, as described in the first embodiment;

[0134] An optimization module is used to optimize the target reception speech based on the target service quality analysis result and put the target service quality analysis result into a service quality evaluation library to optimize the knowledge base based on the service quality evaluation library, as specifically described in Example 1.

[0135] Embodiment 3

[0136] like Figure 5 As shown, based on the above-mentioned digital human interaction method based on multimodal biometric recognition, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Figure 5 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0137] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a digital human interaction program 40 based on multimodal biometrics is stored on the memory 20, and the digital human interaction program 40 based on multimodal biometrics can be executed by the processor 10, thereby realizing the digital human interaction method based on multimodal biometrics in the present application.

[0138] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the digital human interaction method based on multimodal biometric recognition.

[0139] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0140] In one embodiment, when the processor 10 executes the digital human interaction program 40 based on multimodal biometric recognition in the memory 20, the following steps are implemented:

[0141] receiving a target reception signal, and acquiring a target reception speech in a knowledge base based on the target reception signal, wherein the target reception signal is a signal for receiving a target customer;

[0142] When it is identified that the target customer has arrived at the target location, a target scene interaction corresponding to the target location is initiated based on the target reception script;

[0143] Acquire the target customer's interaction status in real time, and perform service quality analysis based on the interaction status to obtain a target service quality analysis result;

[0144] The target reception speech is optimized based on the target service quality analysis result and the target service quality analysis result is put into a service quality evaluation library, so as to optimize the knowledge base based on the service quality evaluation library.

[0145] Embodiment 4

[0146] The present invention also provides a storage medium storing one or more programs, which can be executed by one or more processors to implement the steps of the digital human interaction method based on multimodal biometric recognition described in the above embodiment.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A digital human interaction method based on multimodal biometric recognition, characterized in that: include: receiving a target reception signal, and acquiring a target reception speech in a knowledge base based on the target reception signal, wherein the target reception signal is a signal for receiving a target customer; When it is identified that the target customer has arrived at the target location, a target scene interaction corresponding to the target location is initiated based on the target reception script; Acquire the target customer's interaction status in real time, and perform service quality analysis based on the interaction status to obtain a target service quality analysis result; The target reception speech is optimized based on the target service quality analysis result and the target service quality analysis result is put into a service quality evaluation library, so as to optimize the knowledge base based on the service quality evaluation library.

2. The digital human interaction method based on multimodal biometric recognition according to claim 1, characterized in that: The step of acquiring the target reception speech in the knowledge base based on the target reception signal includes: Acquire target identity information of the target customer based on the target reception signal, wherein the target identity information includes the age and job type of the target customer; The reception speech matching the target identity information in the knowledge base is obtained as the target reception speech.

3. The digital human interaction method based on multimodal biometric recognition according to claim 1, characterized in that: The starting of the target scene interaction corresponding to the target location based on the target reception words includes: Retrieving target interactive content in a knowledge base based on the target position; Explain and ask questions about the product corresponding to the target location based on the target interactive content; The target interactive content is dynamically displayed on a display screen at the target location in real time.

4. The digital human interaction method based on multimodal biometric recognition according to claim 1, characterized in that: The real-time acquisition of the target customer's interaction status includes: Perform voiceprint recognition, speech recognition, emotion recognition and auxiliary recognition on the target customer to collect the target interaction data of the target customer, wherein the target interaction data includes the voiceprint emotion, speech content, emotion expression conclusion and body expression conclusion corresponding to each interaction stage; The interaction status of the target customer is obtained based on the target interaction data.

5. The digital human interaction method based on multimodal biometric recognition according to claim 4 is characterized in that: The performing of service quality analysis based on the interaction status includes: Identifying the proportion of positive emotion expressions corresponding to the target interaction stage based on the target interaction data corresponding to the target interaction stage, and if the proportion of positive emotion expressions is greater than a first preset value, determining that the service quality corresponding to the target interaction stage is excellent; If the proportion of positive emotion expression is greater than a second preset value and less than or equal to the first preset value, then the service quality corresponding to the target interaction stage is determined to be qualified; If the proportion of positive emotion expression is greater than the third preset value and less than or equal to the second preset value, it is determined that the service quality corresponding to the target interaction stage is average; If the proportion of positive emotion expressions is less than or equal to the third preset value, it is determined that the service quality corresponding to the target interaction stage is to be optimized.

6. The digital human interaction method based on multimodal biometric recognition according to claim 1, characterized in that: The optimizing the target reception speech based on the target service quality analysis result includes: Adjusting the target reception speech based on the target service quality analysis result until the service quality of the service quality analysis result is qualified or excellent, and recording the adjusted target reception speech; Matching the target customer's voiceprint information with the adjusted target reception speech, and recording the information in the knowledge base, so that the adjusted target reception speech is automatically matched when providing other customer service to the target customer; The next reception of the target customer is carried out based on the adjusted target reception words.

7. The digital human interaction method based on multimodal biometric recognition according to claim 1, characterized in that: The optimizing the knowledge base based on the service quality evaluation base includes: Acquire a service improvement experience database based on the service quality analysis results corresponding to different identity information in the service quality evaluation database; Based on the service improvement experience database, the explanation speech bias corresponding to different identity information in the knowledge base is optimized and updated.

8. A digital human interaction system based on multimodal biometric recognition, characterized in that: The system comprises: A signal receiving module receives a target reception signal, and acquires a target reception speech in a knowledge base based on the target reception signal, wherein the target reception signal is a signal for receiving a target customer; An interactive module, for initiating a target scene interaction corresponding to the target location based on the target reception script when recognizing that the target customer has arrived at the target location; An analysis module, used to obtain the interaction status of the target customer in real time, and perform service quality analysis based on the interaction status to obtain a target service quality analysis result; The optimization module is used to optimize the target reception speech based on the target service quality analysis result and put the target service quality analysis result into a service quality evaluation library, so as to optimize the knowledge base based on the service quality evaluation library.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a digital human interaction program based on multimodal biometric recognition stored in the memory and executable on the processor. When the digital human interaction program based on multimodal biometric recognition is executed by the processor, the steps of the digital human interaction method based on multimodal biometric recognition as described in any one of claims 1 to 7 are implemented.

10. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the digital human interaction method based on multimodal biometric recognition as described in any one of claims 1-7.

Images