Consultation reply method and device and electronic equipment

Abstract

The invention discloses a consultation reply method and device and electronic equipment. The method comprises the following steps: acquiring audio data consulted by a target object; according to the audio data, determining consultation question data, N target image frames and target voice data; determining first sub-emotion change data according to the N target image frames, and determining second sub-emotion change data according to the target voice data; according to the first sub-emotion change data and the second sub-emotion change data, determining emotion change data of the target object; determining initial reply data corresponding to the consultation question data; and according to the emotion transformation data, adjusting the initial reply data to obtain target reply data. Through application of the method and the device, the problems that the user consultation experience is easy to be poor and the user consultation experience is affected due to the fact that the intelligent virtual person may not pay attention to the emotion change of the user when the virtual person recommends a product to the user in the related technology are solved.

Description

Technical Field

[0001] The present application relates to the field of metaverse, and more specifically, to a consultation reply method, device and electronic device. Background Art

[0002] Regarding the related technologies where virtual humans recommend products to users, since the intelligent virtual humans may not notice changes in user emotions, poor user consultation experience may occur easily. Technical problems that affect the user consultation experience may arise, and no effective solution has been proposed so far. Summary of the invention

[0003] The main purpose of this application is to provide a consultation response method, device and electronic device to solve the problem in the related technology that when a virtual person recommends a product to a user, the intelligent virtual person may not notice the user's emotional changes, which may easily lead to a poor user consultation experience and affect the user consultation experience.

[0004] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a consultation reply method is provided. The method comprises: obtaining audio data of the consultation conducted by the target object; determining the consultation question data, N target image frames, and target voice data based on the audio data, wherein the N target image frames are image frames whose part key point change index of the target object is greater than the first threshold, and the target voice data is data whose tone change index of the target object is greater than the second threshold, and N is a positive integer greater than 1; determining the first sub-emotion change data based on the N target image frames, and determining the second sub-emotion change data based on the target voice data; determining the emotion change data of the target object based on the first sub-emotion change data and the second sub-emotion change data; determining the initial reply data corresponding to the consultation question data; adjusting the initial reply data based on the emotion change data to obtain the target reply data.

[0005] Optionally, determining the first sub-emotion change data based on the N target image frames includes: determining M overall contour data corresponding to the overall contour of the target object and Q local contour data corresponding to the local contour of the target object in the N target image frames, wherein M is a positive integer greater than 1, and Q is a positive integer greater than 1; determining a first contour offset corresponding to the overall contour based on the M overall contour data, and determining a second contour offset corresponding to the local contour based on the Q local contour data; determining an offset difference between the second contour offset and the first contour offset; and determining the first sub-emotion change data based on the offset difference.

[0006] Optionally, determining the second sub-emotion change data based on the target voice data includes: determining voice parameters in the target voice data, wherein the voice parameters include at least: volume parameters and pitch parameters; determining volume change information based on the volume parameters, and determining pitch change information based on the pitch parameters; determining the second sub-emotion change data based on the volume change information and the pitch change information.

[0007] Optionally, the initial reply data is adjusted according to the emotion transformation data to obtain target reply data, including: determining the preferred reply data of the target object; adjusting the initial reply data according to the preferred reply data and the emotion transformation data to obtain target reply data.

[0008] Optionally, the initial reply data is adjusted according to the emotion transformation data to obtain target reply data, including: determining the language data of the target object in the audio data, wherein the language data includes sentence data and vocabulary data; determining target reply data according to the language data; adjusting the initial reply data according to the target reply data and the emotion transformation data to obtain target reply data.

[0009] Optionally, determining initial reply data corresponding to the consulting question data includes: determining a reply type to the consulting question data; when the reply type is a selection type, obtaining preference behavior data of the target object, wherein a correlation index between the preference behavior data and the consulting question data is greater than a third threshold; and determining the initial reply data based on the preference behavior data.

[0010] Optionally, after adjusting the initial reply data according to the emotion change data and obtaining the target reply data, it also includes: generating virtual human behavior parameters according to the audio data; driving the virtual human to perform actions according to the virtual human behavior parameters, and replying to the target reply data.

[0011] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a consultation reply device is provided. The device includes: an acquisition module, which is used to acquire audio data of the consultation conducted by the target object; a first determination module, which is used to determine the consultation question data, N target image frames, and target voice data based on the audio data, wherein the N target image frames are image frames whose part key point change index of the target object is greater than the first threshold, and the target voice data is data whose tone change index of the target object is greater than the second threshold, and N is a positive integer greater than 1; a second determination module, which is used to determine the first sub-emotion change data based on the N target image frames, and determine the second sub-emotion change data based on the target voice data; a third determination module, which is used to determine the emotion change data of the target object based on the first sub-emotion change data and the second sub-emotion change data; a fourth determination module, which is used to determine the initial reply data corresponding to the consultation question data; and an adjustment module, which is used to adjust the initial reply data based on the emotion change data to obtain the target reply data.

[0012] Optionally, the device provided in the embodiment of the present application also includes an offset determination module, which is used to determine M overall contour data corresponding to the overall contour of the target object and Q local contour data corresponding to the local contour of the target object in N target image frames, wherein M is a positive integer greater than 1, and Q is a positive integer greater than 1; determine a first contour offset corresponding to the overall contour based on the M overall contour data, and determine a second contour offset corresponding to the local contour based on the Q local contour data; determine an offset difference between the second contour offset and the first contour offset; and determine first sub-emotion change data based on the offset difference.

[0013] Optionally, the device provided in the embodiment of the present application also includes a speech determination module for determining speech parameters in the target speech data, wherein the speech parameters include at least: volume parameters and pitch parameters; determining volume change information based on the volume parameters, and determining pitch change information based on the pitch parameters; determining second sub-emotion change data based on the volume change information and the pitch change information.

[0014] Optionally, the device provided in the embodiment of the present application also includes a preference determination module for determining the preference reply data of the target object; based on the preference reply data and the emotion change data, the initial reply data is adjusted to obtain target reply data.

[0015] Optionally, the device provided in the embodiment of the present application also includes a language determination module, which is used to determine the language data of the target object in the audio data, wherein the language data includes sentence data and vocabulary data; based on the language data, target reply data is determined; based on the target reply data and the emotion transformation data, the initial reply data is adjusted to obtain the target reply data.

[0016] Optionally, the device provided in the embodiment of the present application also includes a reply type determination module for determining the reply type to the consulting question data; when the reply type is a selection type, obtaining the preference behavior data of the target object, wherein the correlation index between the preference behavior data and the consulting question data is greater than a third threshold; and determining the initial reply data based on the preference behavior data.

[0017] Optionally, the device provided in the embodiment of the present application also includes a behavior determination module, which is used to generate virtual human behavior parameters based on the audio data; drive the virtual human to perform actions based on the virtual human behavior parameters, and reply to the target response data.

[0018] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a computer-readable storage medium is provided, and the readable storage medium includes: the computer-readable storage medium includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned consultation reply methods.

[0019] In order to achieve the above-mentioned purpose, according to another aspect of the present application, an electronic device is provided, which includes: a memory storing an executable program; and a processor for running the program, wherein the program executes any one of the above-mentioned consultation reply methods when running.

[0020] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a computer program product is provided, which includes: computer instructions, which, when executed by a processor, implement the steps of any one of the above-mentioned consultation reply methods.

[0021] In an embodiment of the present application, by acquiring audio data of the target object for consultation; according to the audio data, determining the consultation question data, N target image frames, and target voice data, wherein the N target image frames are image frames whose key point change index of the target object is greater than the first threshold, and the target voice data is data whose tone change index of the target object is greater than the second threshold, and N is a positive integer greater than 1; according to the N target image frames, determining the first sub-emotion change data, and according to the target voice data, determining the second sub-emotion change data; according to the first sub-emotion change data and the second sub-emotion change data, determining the emotion change data of the target object; determining the initial reply data corresponding to the consultation question data; according to the emotion change data, adjusting the initial reply data to obtain the target reply data. By acquiring the audio data when the target object is consulting, and extracting the consultation question data, N target image frames, and target voice data from the audio data. Then, for the N target image frames, determining the first sub-emotion change data, for the target voice data, determining the second sub-emotion change data, and combining the first sub-emotion change data and the second sub-emotion change data, the overall emotion change data of the target object is obtained. At the same time, according to the initial reply data corresponding to the consultation question data. Finally, based on the determined emotional change data, the initial response data is adjusted and optimized to ensure that the response content is more in line with the target object's emotional state at the time, thereby providing a personalized response. After such adjustments, target response data for the target object's specific questions and emotional state will eventually be generated, thereby solving the problem in related technologies that when virtual humans recommend products to users, the intelligent virtual humans may not notice the changes in user emotions, which can easily lead to poor user consultation experience and technical problems that affect user consultation experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0023] Figure 1 It is a hardware structure block diagram of a computer terminal (or mobile device) for implementing the consultation reply method;

[0024] Figure 2 is a flowchart of a consultation reply method according to Example 1 of the present application;

[0025] Figure 3 is a schematic diagram of a consultation reply device provided according to an embodiment of the present application;

[0026] Figure 4 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] It should be noted that the collected information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with relevant laws, regulations and standards, necessary confidentiality measures are taken, and public order and good customs are not violated, and corresponding operation entrances are provided for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions to provide users with corresponding operation entrances for users to choose to agree or refuse the results of automated decision-making; if the user chooses to refuse, the expert decision-making process will be entered.

[0030] Example 1

[0031] According to an embodiment of the present application, an embodiment of a consultation reply method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0032] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1It is a hardware structure block diagram of a computer terminal (or mobile device) for implementing the consultation reply method. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0033] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0034] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the consultation reply method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned consultation reply method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0035] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0036] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0037] Under the above operating environment, this application provides Figure 2 The following is the method of replying to the inquiry: Figure 2 It is a flowchart of the consultation response method according to Example 1 of the present application.

[0038] Step S202, obtaining audio data of the target object's consultation;

[0039] In step S202 provided in the present application, audio data of the target object's consultation is obtained.

[0040] Among them, the target object is involved, and the target object is used to represent the object to interact with the virtual person. For example, when interacting with the virtual person, the target object can be any individual who needs to communicate.

[0041] Among them, audio data is involved, and the audio data is the data related to voice and video generated by the target object during the communication process with the virtual person. The audio data may include voice data and video data.

[0042] By capturing the target object's consultation audio data, it can provide a data basis for subsequent virtual humans to analyze the target object's consultation content and emotional changes, thereby helping the intelligent virtual humans to interact according to the target object's emotional changes, and further helping to improve the target object's consultation experience.

[0043] When obtaining the audio data of the target object for consultation, it is possible to consider integrating speech recognition technology and emotion perception algorithms. It should be noted that the method for obtaining the audio data of the target object for consultation is not specifically limited here, and can be customized according to actual applications and scenarios.

[0044] Step S204, determining consulting question data, N target image frames, and target voice data based on the audio data, wherein the N target image frames are image frames whose key point variation index of the target object is greater than a first threshold, and the target voice data are data whose tone variation index of the target object is greater than a second threshold, and N is a positive integer greater than 1;

[0045] In step S204 provided in the present application, consulting question data, N target image frames, and target voice data are determined.

[0046] Among them, the consulting question data is involved, which is data extracted from the audio data of the target object and used to describe the specific content of the consulting question. For example, if the target object asks "How is the weather today?", the corresponding consulting question data includes "How is the weather today?"

[0047] Among them, target image frames are involved, which are obtained based on audio data, captured during the interaction between the target object and the virtual human, and used to help analyze the target object's emotional changes, and are image frames with significant expression or movement changes in specific body parts of the target object. For example, in video data, when the target object nods or changes facial expression, these image frames are marked as target image frames.

[0048] Among them, target voice data is involved, which is obtained based on audio data and captured during the interaction between the target object and the virtual human, and the target object's voice features, such as tone, intonation, speed, etc., have obvious changes in audio data. For example, when the target object's tone suddenly changes from a calm tone to a hurried or angry tone, the corresponding voice data is the target voice data.

[0049] Among them, the part key point change index is involved, which is an index used to quantify the degree of change of key points of specific parts of the target object's body (such as face and limbs) in consecutive image frames. For example, the part key point change index can be the change range of the target object's eyebrow lifting range, the curvature of the corner of the mouth, etc.

[0050] Among them, the first threshold is involved, which is the critical value used to judge whether the changes in the key points of the parts are significant. For example, when the arc of the smile at the corner of the mouth rises by more than 5%, it can be considered that the facial expression of the target object has changed significantly. At this time, 5% is the first threshold.

[0051] Among them, the tone change index is involved, which is an index used to quantify the degree of tone change in the target object's voice. For example, the tone change index can be the degree of change of the target object's tone from calm to impatient.

[0052] Among them, the second threshold is involved, which is a critical value used to determine whether the tone change in the target object's voice has reached a level that requires special attention. For example, when the tone change index in the target object's voice exceeds 20%, it is considered that the user's emotions have fluctuated significantly, and 20% at this time is the second threshold.

[0053] By determining the consultation question data, N target image frames, and target voice data based on the audio data, it helps to provide a data basis for the subsequent comprehensive analysis of the target object's consultation content, facial expression changes, and tone fluctuations, thereby helping to accurately understand the target object's consultation intentions and emotional state, thereby helping to solve the problem of ignoring user emotions and non-verbal information when only relying on text information for consultation processing in subsequent steps, thereby being able to understand user needs more comprehensively and accurately and avoid poor user experience caused by misunderstandings.

[0054] When determining consulting question data, N target image frames, and target voice data based on audio data, it is possible to consider integrating multi-dimensional data analysis capabilities and machine learning algorithms. For example, the virtual person can adjust the first threshold and the second threshold based on the historical consulting records of the target object and current environmental information, such as time, location, weather, etc., to more accurately determine the consulting question data, N target image frames, and target voice data of the target object. It should be noted that there is no specific limitation on the method of determining consulting question data, N target image frames, and target voice data, and can be customized according to actual applications and scenarios.

[0055] Step S206, determining first sub-emotion change data according to the N target image frames, and determining second sub-emotion change data according to the target voice data;

[0056] In step S206 provided in the present application, the first sub-emotion change data and the second sub-emotion change data are determined.

[0057] Among them, the first sub-emotion change data is involved, which is the emotional state change data inferred based on the changes in the key points of the target object captured in the target image frame. For example, if several consecutive frames in the target image frame show that the target object changes from smiling to frowning, then this may indicate that the user's emotion changes from happiness to displeasure or confusion. Correspondingly, this emotional change information from happiness to displeasure is the first sub-emotion change data.

[0058] Among them, the second sub-emotion change data is involved, which is obtained based on the target voice data and is used to reflect the emotional state change data inferred from the changes in the target object's tone, intonation, speech speed and other characteristics. For example, if the target object suddenly changes from a calm tone to a hurried tone or increases the volume, this may indicate that the user's emotion changes from calm to impatient or excited. Correspondingly, this emotional change information from calm to impatient is the second sub-emotion change data.

[0059] Compared with relying solely on voice or image, by determining the first sub-emotion change data and the second sub-emotion change data based on the target image frame and the target voice data, a comprehensive analysis of the data of two different modalities, image and voice, is achieved, which helps the virtual person to more keenly capture the emotional fluctuations of the target object, and to comprehensively judge the emotional changes of the target object, avoiding communication barriers caused by inaccurate emotion recognition. For example, the virtual person can adjust the dialogue strategy in real time according to the emotional changes of the target object; when the target object is depressed, the virtual person can adopt a more gentle and encouraging tone to communicate; when the target object seems impatient, the virtual person can speed up the answer and use concise and clear language. Furthermore, by separately determining the first sub-emotion change data and the second sub-emotion change data, more comprehensive and accurate emotion recognition results can be provided, which helps to create a more natural and smooth interactive environment, and thus helps to improve user satisfaction.

[0060] In actual applications, the target object's emotions can be analyzed from more dimensions by further combining historical behavior data and social network information. For example, by understanding the target object's recent life events, the virtual person can better understand the reasons for the target object's emotional changes and provide more considerate and appropriate responses. In addition, the virtual person can also gradually master more emotion recognition and emotion management skills through continuous learning, such as identifying specific micro-expressions and learning the target object's personality characteristics, so as to adapt to more complex and diverse emotional expressions.

[0061] It should be noted that the method for determining the first sub-emotion change data and the second sub-emotion change data is not specifically limited here, and can be customized according to actual applications and scenarios.

[0062] Step S208, determining the emotion change data of the target object according to the first sub-emotion change data and the second sub-emotion change data;

[0063] In step S208 provided in the present application, the emotion change data of the target object is determined.

[0064] Among them, the emotion change data is involved, which is obtained by comprehensively analyzing the first sub-emotion change data and the second sub-emotion change data, and is used to represent the change state of the target object's emotional state. For example, compared with the initial stage, the target object frowns (first sub-emotion change data) and the tone of voice rises (second sub-emotion change data) during the communication process, then the emotion change data may describe the target object's emotion gradually changing from calm to impatient.

[0065] By determining the target object's emotion transformation data based on the first sub-emotion change data and the second sub-emotion change data, the limitation that a single data (such as only facial expressions or only voice features) cannot fully reflect the target object's true emotional state is avoided. The target object's emotion transformation data combines the first sub-emotion change data and the second sub-emotion change data, so that the virtual human can not only obtain the emotion information of the key point changes, but also obtain the emotion information of the tone changes, thereby improving the accuracy and reliability of emotion recognition, and then by accurately capturing and responding to the user's emotional changes, the interaction mode between the virtual human and the target object can be adjusted to improve the user experience and satisfaction.

[0066] In the process of determining the emotion change data, in addition to integrating the first sub-emotion change data and the second sub-emotion change data, the user's historical behavior data, conversation context, and even the user's social and cultural background and other multi-dimensional information can be further considered to enhance the depth and accuracy of emotion recognition. For example, the system can predict the user's emotional tendencies under specific topics or situations by learning the user's previous emotional response patterns, so as to take corresponding preventive measures at the beginning of the conversation to avoid triggering the user's negative emotions. In addition, the virtual person can also understand the triggering factors of the user's emotional changes through the analysis of emotion change data, provide emotion management education or suggestions, help users better control and express their emotions, and increase the role of the virtual person in emotional support and management.

[0067] It should be noted that the method for determining the emotion change data of the target object is not specifically limited here, and can be customized according to actual applications and scenarios.

[0068] Step S210, determining initial answer data corresponding to the consultation question data;

[0069] In step S210 provided in the present application, initial response data corresponding to the consultation question data is determined.

[0070] Among them, the initial response data is involved, which is the preliminary response data determined based on the consultation question data. The initial response data can directly and accurately answer the questions asked by the target object, or solve the initial needs of the target object. For example, if the target object asks "What should I pay attention to in my recent health condition", the corresponding initial response data can be: "Given your age and past medical history, you should pay special attention to blood pressure and sleep quality recently. It is recommended to exercise moderately and maintain a healthy diet."

[0071] Based on the consultation question data, the virtual person can quickly identify and understand the specific consultation content of the target object, and instantly generate or retrieve highly relevant and accurate initial response data, clarify the basic direction and content of the conversation, and achieve a response to the target object's consultation content, providing a basis for emotional adaptive adjustment for the subsequent final response, which in turn helps to help users solve problems or meet needs.

[0072] It should be noted that there is no specific limitation on the method of determining the initial response data corresponding to the consultation question data, and the settings can be customized according to the actual application and scenario.

[0073] Step S212, adjusting the initial response data according to the emotion transformation data to obtain the target response data.

[0074] In step S212 provided in the present application, the initial response data is adjusted to obtain the target response data.

[0075] Among them, the target response data is involved. The target response data is obtained based on the initial response data and adjusted according to the emotion change data, and is used to feed back to the current target object. For example, if the target object inquires about the detailed information of the product, and the emotion change data shows that the target object is currently anxious, the target response data may include: "I fully understand your concerns about the product. Let me explain the product's safety performance and warranty policy in detail. You can rest assured that we have 24-hour customer service support to answer any questions you have at any time."

[0076] Based on the initial reply data and taking into account the user's emotion change data, the target reply data is obtained. This process optimizes the initially generated or retrieved reply information by taking the emotion change data as the basis for adjustment, which not only ensures the accuracy of the reply to the target object's consulting questions, but also matches the current state of the target object at the emotional level, that is, it takes into account the target object's current emotional state, thereby providing a more personalized and emotional reply, and effectively solves the problem in related technologies that when virtual humans recommend products to users, the intelligent virtual humans may not notice the changes in user emotions, which may easily lead to poor user consultation experience and technical problems that affect the user consultation experience.

[0077] When generating target response data, the virtual human intelligent system can also combine the user's historical consultation records, personal preferences, social and cultural background and other multi-dimensional information to further refine and optimize the response strategy. For example, if the system knows that the user prefers concise and clear explanations in past consultations, even if the emotional change data shows that the current user is anxious, the virtual human can use a more direct and concise expression method while ensuring the integrity of the information, and at the same time convey a caring and supportive attitude through the adjustment of the tone, so as to achieve the dual effect of solving the problem and soothing the emotions. In addition, the virtual human can also accumulate the correlation data between the user's emotions and the response strategy through the generation of target response data, continuously optimize its emotion recognition and response algorithms, improve the system's emotional intelligence and service efficiency, and provide users with a more natural, smooth and humane communication experience.

[0078] It should be noted that there is no specific limitation on the method of adjusting the initial response data to obtain the target response data, and the settings can be customized according to the actual application and scenario.

[0079] Optionally, in the consultation response method provided in the embodiment of the present application, the first sub-emotion change data is determined based on N target image frames, including: determining M overall contour data corresponding to the overall contour of the target object and Q local contour data corresponding to the local contour of the target object in the N target image frames, wherein M is a positive integer greater than 1, and Q is a positive integer greater than 1; determining a first contour offset corresponding to the overall contour based on the M overall contour data, and determining a second contour offset corresponding to the local contour based on the Q local contour data; determining the offset difference between the second contour offset and the first contour offset; and determining the first sub-emotion change data based on the offset difference.

[0080] In this embodiment, specific steps of determining the first sub-emotion change data based on N target image frames are described.

[0081] Among them, the overall contour data is involved. The overall contour data is extracted from the target image frame and is used to reflect the overall contour of the target object. It can be the change data of the overall position, size, and direction of the target object's head, face, etc. For example, the overall contour can be the head. If the target object turns its head or shakes its head left and right during the conversation, the overall contour data will record the change of the head position.

[0082] Among them, local contour data is involved. The local contour data is extracted from the target image frame and is used to reflect the changes in local features in specific parts of the target object (such as the head, face, etc.). For example, the specific part can be the face, and the local feature changes in the face can be subtle changes in the movement of the eyes, mouth, and eyebrows. If the eyebrows of the target object gradually rise in several image frames, the changes in the eyebrows will be recorded according to the local contour data.

[0083] Among them, the first contour offset is involved, which is determined by M overall contour data and is used to reflect the offset of the overall contour change of the target object. The first contour offset can be used to analyze the overall specific part of the target object during the conversation, such as the behavior change of the head.

[0084] Among them, the second contour offset is involved, which is determined by Q local contour data and is used to reflect the offset of the local feature change in a specific part of the target object. For example, by analyzing the position changes of the target object's eyes, mouth, eyebrows and other feature points in N target image frames, the second contour offset can be obtained to determine whether the user expresses specific emotional changes through local expressions such as smiling and frowning.

[0085] Among them, the offset difference is involved. The offset difference is used to reflect the consistency or difference between the overall behavior and local expression changes of the target object, and is obtained based on the difference between the first contour offset and the second contour offset. For example, if the target object's head remains relatively stable, but the eyebrows gradually rise in several image frames, based on the large offset difference, it can be indicated that the target object may have a change in surprise or questioning emotions. For another example, if the target object's head moves to the left, and the eyebrows also show a move to the left, then the offset difference is small at this time, which can indicate that the target object only moves its head without any change in emotion.

[0086] In the steps involved in this embodiment, M overall contour data corresponding to the overall contour of the target object are identified and extracted from N target image frames. Here, M is a positive integer greater than 1, which means that more than one overall contour data is extracted. At the same time, Q local contour data corresponding to the local contour of the target object are extracted from the N target image frames. Q is also a positive integer greater than 1, indicating that multiple local contour data are identified. Based on the extracted M overall contour data, the first contour offset corresponding to the overall contour is determined. Based on the Q local contour data, the second contour offset corresponding to the local contour is determined. After determining the first contour offset and the second contour offset, the offset difference is determined based on the difference between the first contour offset and the second contour offset. Finally, based on the offset difference, the first sub-emotion change data is further analyzed and determined.

[0087] By comprehensively analyzing the specific parts of the target object as a whole (such as the head, face, etc.) and the local feature changes in the specific parts, the complex emotional changes of the target object can be accurately captured. By determining the offset difference based on the first contour offset and the second contour offset, the influence of the overall change of the specific part (such as the sudden movement of the face to the left) on the virtual human's analysis of the local feature changes in the specific parts of the target object is avoided. Therefore, it can be determined that the local feature changes in the specific part are caused by the emotional changes of the target object, rather than the overall changes of the specific part. Therefore, the emotional changes of the target object can be captured more accurately, which helps to provide more personalized and professional interactions and improve the user consultation experience.

[0088] In practical applications, in addition to calculating the offset difference to determine the first sub-emotion change data, other emotion-related parameters, such as voice frequency modulation, speech speed changes, continuity and coherence of user behavior, etc., can also be combined for multi-dimensional analysis to improve the accuracy and comprehensiveness of emotion recognition. In addition, taking into account different cultures and individual differences, the system can also establish a more detailed emotion recognition model through training and learning, such as distinguishing different types of smiles (happy, embarrassed, sarcastic, etc.), as well as the typical emotional expression characteristics of different age and gender groups, so as to provide more targeted and culturally sensitive services in a wider range of scenarios.

[0089] Optionally, in the consultation response method provided in the embodiment of the present application, second sub-emotion change data is determined based on the target voice data, including: determining voice parameters in the target voice data, wherein the voice parameters include at least: volume parameters and pitch parameters; determining volume change information based on the volume parameters, and determining pitch change information based on the pitch parameters; determining the second sub-emotion change data based on the volume change information and the pitch change information.

[0090] In this embodiment, specific steps of determining the second sub-emotion change data based on the target speech data are described.

[0091] Among them, voice parameters are involved, which are parameters determined based on target voice data and used to reflect the voice state of the target object. By analyzing the voice parameters, the emotional state of the target object can be indirectly obtained.

[0092] Among them, a volume parameter is involved, which is a parameter used to reflect the volume of the target object. For example, when the target object is expressing excitement, the corresponding speaking volume will increase significantly.

[0093] Among them, the pitch parameter is involved, which is a parameter used to reflect the pitch characteristics of the target object's voice, including the fundamental frequency (pitch) of the pitch, the range of pitch change, and the trend of pitch change. For example, when the pitch parameter shows that the fundamental frequency of the pitch is increased, the pitch change range is wide, and the trend of pitch change is rising, and the volume parameter shows that the volume is increased, then it can be judged that the target object is in an excited or agitated state.

[0094] Among them, the volume change information is involved, which is determined based on the volume parameter and is used to reflect the volume change of the target object during the conversation. For example, by analyzing the volume of the user's consecutive sentences, it can be determined whether the volume gradually increases or decreases. This change information is helpful to judge the fluctuation of the user's emotions, such as the change from calm to excited.

[0095] Among them, pitch change information is involved, which is determined based on pitch parameters and is used to reflect the pitch change of the target object during the conversation.

[0096] Determine the second sub-emotion change data based on the target voice data, including: determine the voice parameters in the target voice data, wherein the voice parameters include at least: volume parameters and pitch parameters; determine the volume change information based on the volume parameters, and determine the pitch change information based on the pitch parameters; determine the second sub-emotion change data based on the volume change information and the pitch change information.

[0097] In the steps involved in this embodiment, first, a speech parameter reflecting the speech state of the target object is extracted from the target speech data. The speech parameter includes at least a volume parameter and a pitch parameter. Then, based on the extracted volume parameter, the volume change information is analyzed and determined. At the same time, based on the pitch parameter, the pitch change information is analyzed and determined. After obtaining the volume change information and the pitch change information, the second sub-emotion change data is determined by combining the two types of information.

[0098] The speech parameters reflecting the speech state of the target object are extracted from the target speech data, including at least volume parameters and pitch parameters, and the corresponding volume change information and pitch change information are determined according to the volume parameters and pitch parameters respectively. The second sub-emotion change data of the target object is identified and determined by integrating the volume change information and the pitch change information, avoiding the limitation of relying solely on facial expressions or body language to recognize emotions. By deeply analyzing the slight changes in the speech, the virtual human can capture deeper information about the target object's emotions. In particular, when the target object's face is partially blocked or the body posture changes are not obvious, speech analysis can serve as an important supplementary means of emotion recognition, thereby improving the accuracy and comprehensiveness of emotion recognition and enhancing the user consultation experience.

[0099] In addition to volume and pitch parameters, speech analysis can also take into account more details, such as the rhythm of the speech, speaking speed, pauses, the use of modal particles, etc., which may reflect the user's emotional state. For example, a fast speaking speed may be associated with anxiety or excitement, while a slow speaking speed may indicate that the user is tired. In addition, the virtual human can also combine semantic understanding to analyze the consistency between the user's speech content and emotional changes, further improving the accuracy of emotion recognition.

[0100] Optionally, in the consultation response method provided in the embodiment of the present application, the initial response data is adjusted based on the emotion change data to obtain the target response data, including: determining the preference response data of the target object; adjusting the initial response data based on the preference response data and the emotion change data to obtain the target response data.

[0101] In this embodiment, specific steps of adjusting the initial response data according to the emotion transformation data to obtain the target response data are described.

[0102] Among them, preference reply data is involved. The preference reply data is obtained based on the target object's past behavior, interaction records, personal preferences and other information, and is used to reflect the target object's preferred content, style or tone of reply data. For example, if the target object prefers a relaxed and humorous way of communication, the preference reply data may contain more humorous elements or relaxed interjections; conversely, if the target object prefers formal and professional conversations, the preference reply data will tend to use professional terms and formal tones.

[0103] In the steps involved in this embodiment, first, the preferred reply data of the target object is determined, that is, the type or style of reply that the target object likes or expects is understood. Then, based on these preferred reply data and the emotion change data, the initial reply data is adjusted so that the reply takes into account the preferences of the target object and the emotion of the target object at the same time, and finally, the adjusted target reply data is obtained.

[0104] By determining the target object's preferred response data, the user's preferred communication method can be used to provide personalized responses based on the target object's preferred content, style or tone. By adjusting the initial response data based on the preferred response data and the emotional transformation data, the target response data obtained after the adjustment not only takes into account the target object's communication preferences, but also the target object's emotions, so that the virtual person can not only understand the user's needs, but also respond in a way that is closer to the user's emotions and communication preferences, reducing interaction barriers, improving dialogue efficiency, and enhancing the user's experience.

[0105] In addition to real-time emotion change data and preference response data, virtual humans can also combine other factors, such as the user's cultural background, social role, current situation, etc., to further refine their response strategies. For example, in a business environment, you can prefer to use professional terms and formal tones, while when interacting with children, you can use simpler, more vivid and friendly expressions. In addition, virtual humans can also continuously optimize the accuracy of their understanding of user emotion changes and preferences through learning and feedback mechanisms, thereby providing more precise and personalized services.

[0106] Optionally, in the consultation response method provided in the embodiment of the present application, the initial response data is adjusted based on the emotion change data to obtain target response data, including: determining the language data of the target object in the audio data, wherein the language data includes sentence data and vocabulary data; determining the target response data based on the language data; adjusting the initial response data based on the target response data and the emotion change data to obtain the target response data.

[0107] In this embodiment, specific steps of adjusting the initial response data according to the emotion transformation data to obtain the target response data are described.

[0108] Among them, language data is involved, which is determined based on audio data and is used to reflect the language characteristics of the target object. The language data includes sentence data and word data. For example, the target object tends to use inverted sentences and dialects.

[0109] Among them, sentence data is involved, which is the data about sentence structure and grammatical features in language data, involving the sentence patterns used by the target object in the conversation, such as declarative sentences, interrogative sentences, exclamatory sentences, inverted sentences, etc. Sentence data can reflect the language style, habits, attitude and intention of the target object. For example, the target object may use interrogative sentences to express uncertainty about the information or seek explanation.

[0110] Among them, word usage data is involved, which is the data in the language data about the words and phrases selected by the target object in the conversation and their frequency of use, including the selection of specific words and the use of emotional words. Word usage data not only reflects the language style and habits of the target object, for example, the target object likes to communicate in Cantonese, but also indirectly reveals the emotional state of the target object. For example, the use of words such as "very" and "extremely" that strengthen the tone may indicate that the target object is emotionally excited, while the use of vague words such as "maybe" and "probably" may indicate that the target object is uncertain.

[0111] Among them, target response data is involved, and the target response data is response data determined based on language data, and reflects the language style habits (such as sentence habits and word usage habits) of the target object.

[0112] In the steps involved in this embodiment, first, the language data used by the target object is identified and extracted from the audio data, including sentence data and word data. Sentence data may involve the structure and grammatical features of the sentence, while word data focuses on the vocabulary, phrases and their frequency of use selected by the target object when expressing. Subsequently, based on the language data, the target response data that conforms to the language style and habits of the target object is determined. On the basis of determining the target response data, the emotional transformation data is considered, and the initial response data is appropriately adjusted and optimized to make it more in line with the current emotional state of the target object. After such adjustments, the target response data is finally obtained.

[0113] By identifying and extracting the language data used by the target object from the audio data, including sentence data and word data. Sentence data analysis can understand the target object's expression habits, language style and possible emotional tendencies, which helps to more accurately grasp the context and intention of the conversation and provide a basis for the subsequent generation of responses that conform to their expression methods. Word data analysis can understand the target object's vocabulary preferences and possible cultural background, which helps to understand their intentions more accurately, while avoiding the use of words that may cause misunderstandings or discomfort, and improving the fluency and comfort of the conversation. After mastering the target object's sentence data and word data, it is possible to more accurately understand the target object's language expression habits, and then determine the target response data that conforms to its language characteristics. Through the target response data and emotional transformation data, the initial response data is adjusted, and the target response data obtained can not only meet the target object's response needs, but also resonate with the target object in terms of language style and emotional state, thereby helping to achieve smoother and more efficient communication, and thus helping to enhance the user consultation experience.

[0114] In addition to emotion change data, sentence structure data, and word usage data, the virtual human intelligent dialogue system can also combine more dimensional information, such as the user's social role, cultural background, dialogue context, etc., to further refine its response strategy. For example, when dealing with cross-cultural communication, it is also necessary to consider the acceptance of different cultures for specific emotional expressions and language styles.

[0115] Optionally, in the consultation response method provided in the embodiment of the present application, determining the initial reply data corresponding to the consultation question data includes: determining the reply type to the consultation question data; when the reply type is a selection type, obtaining the preference behavior data of the target object, wherein the correlation index between the preference behavior data and the consultation question data is greater than a third threshold; and determining the initial reply data based on the preference behavior data.

[0116] In this embodiment, specific steps of determining initial answer data corresponding to consulting question data are described.

[0117] Among them, the answer type is involved, which is the answer method or format category used by the virtual person when answering the consulting questions of the target object. The answer type may include but is not limited to explanation type, suggestion type, instruction type, selection type, confirmation type, etc. For example, when the user asks about the product function, the virtual person can give an explanation type answer and introduce the product features in detail, and when the user needs guidance, the virtual person can give a suggestion type answer.

[0118] Among them, the selection type is involved, which is a response form that requires the target object to choose from multiple options. For example, when the target object asks which product is more suitable for him or her, the virtual person can give several recommended options and let the target object choose according to his or her own needs or preferences, instead of giving a definite answer directly.

[0119] Among them, preference behavior data is involved. The preference behavior is obtained based on the target object's past behavior, interaction records, personal preferences and other information, and is used to reflect the target object's preference behavior. The preference behavior data may include the target object's tendency to prefer a certain product, service, or solution, or the target object's tendency behavior when faced with a choice. For example, when the target object buys electronic products, it is more inclined to choose high-end brands, so "high-end brand tendency" is the user's preference behavior data.

[0120] Among them, the correlation index is involved, which is an index used to quantify the degree of association between the preference behavior data and the consultation question data. The larger the value of the correlation index, the closer the connection between the preference behavior data and the consultation question data is, and the more likely it is to affect the choice of the target object.

[0121] Among them, a third threshold is involved, which is a threshold used to determine whether the degree of correlation between the preference behavior data and the consultation question data is close, so as to determine whether it is necessary to adjust the initial response data based on the preference behavior data.

[0122] In the steps involved in this embodiment, when processing the consulting question data, the answer type corresponding to the consulting question data must first be determined. If the answer type is determined to be a selection type (i.e., one or more answers are selected from the preset options), the target object's preference behavior data is further obtained. The correlation index between the preference behavior data and the consulting question data needs to exceed a third threshold value to ensure that the selected preference behavior data can accurately reflect the target object's potential preference behavior for the consulting question, and to determine the initial answer data based on the obtained preference behavior data.

[0123] By determining the type of response to the consulting question data (for example, explanatory, suggestive, directive, selective, and confirmatory), more targeted responses can be made based on the consulting questions of the target object, enhancing the matching of the problem domain response and ensuring that personalized responses can be provided based on user needs and preferences. When the response type is a selection type, by considering the target object's preference behavior data, the response can be closer to the target object's expectations and needs, thereby improving the target object's satisfaction and experience. By using the preference behavior data that is highly relevant to the target question, the response content can more accurately reflect the target object's true intentions and potential needs, reducing misunderstandings and deviations. The initial response data determined based on the preference behavior data is more in line with the needs of the target object, which helps to improve user satisfaction and participation, allowing users to feel understood and respected during the consultation process, thereby improving service experience and trust.

[0124] In addition, through dynamic learning and optimization, the third threshold can be continuously adjusted and refined to more accurately determine which preference behavior data has a substantial impact on the current consultation issue and which can be ignored to achieve more accurate responses.

[0125] Optionally, in the consultation response method provided in the embodiment of the present application, the initial response data is adjusted based on the emotion change data to obtain the target response data, and it also includes: generating virtual human behavior parameters based on the audio data; driving the virtual human to perform actions based on the virtual human behavior parameters, and replying to the target response data.

[0126] In this embodiment, the specific steps after adjusting the initial response data according to the emotion transformation data to obtain the target response data are described.

[0127] Among them, virtual human behavior parameters are involved, which are parameters used to control the virtual human to display behaviors such as facial expressions, body movements, and changes in voice.

[0128] Among them, virtual humans are involved. Virtual humans are digital characters that can simulate human behavior and language interaction. They can be created through computer graphics and artificial intelligence technology. For example, virtual humans can express friendliness and understanding through actions such as smiling and nodding.

[0129] In the steps involved in this embodiment, after obtaining the target reply data, the audio data is further used to generate virtual human behavior parameters. These parameters describe in detail the specific behaviors that the virtual human should show when replying to the user, including but not limited to facial expressions, body movements, head rotation, etc., so that the virtual human's performance is more vivid and natural, and consistent with the reply content and emotional state. According to the virtual human behavior parameters, the virtual human will make a series of actions, expressions, etc. that match the reply content while replying to the target reply data.

[0130] After obtaining the target reply data, the virtual human behavior parameters generated by combining the audio data are driven by the virtual human behavior parameters, so that the virtual human can show rich facial expressions, body movements and voice rhythms when replying to the user, and are consistent with the reply content and emotional state, thereby enhancing the expressiveness and credibility of the virtual human, achieving the synchronization of the virtual human's emotional expression with the user's emotional state, improving the realism and humanity of the conversation, and allowing users to experience a more intimate and real interactive experience.

[0131] In addition to generating virtual human behavior parameters based on emotion change data, more contextual information can be combined, such as the user's personal preferences, cultural background, and conversation scenarios, to further enrich and customize the virtual human's behavior display. For example, in interactions with children, virtual humans can show more lively and friendly expressions and body movements, and use simpler and more imaginative language; in professional consulting scenarios, virtual humans can show more formal and focused appearance behaviors, and use professional terms in specific fields.

[0132] The consultation reply method provided by the embodiment of the present application is obtained by obtaining audio data of the consultation conducted by the target object; according to the audio data, the consultation question data, N target image frames, and target voice data are determined, wherein the N target image frames are image frames whose change index of the key points of the target object is greater than the first threshold value, and the target voice data is data whose tone change index of the target object is greater than the second threshold value, and N is a positive integer greater than 1; according to the N target image frames, the first sub-emotion change data is determined, and according to the target voice data, the second sub-emotion change data is determined; according to the first sub-emotion change data and the second sub-emotion change data, the emotion change data of the target object is determined; the initial reply data corresponding to the consultation question data is determined; according to the emotion change data, the initial reply data is adjusted to obtain the target reply data. By obtaining the audio data when the target object is consulting, and extracting the consultation question data, N target image frames, and target voice data from the audio data. Then, for the N target image frames, the first sub-emotion change data is determined, for the target voice data, the second sub-emotion change data is determined, and the first sub-emotion change data and the second sub-emotion change data are combined to obtain the overall emotion change data of the target object. At the same time, according to the initial reply data corresponding to the consultation question data. Finally, according to the determined emotion change data, the initial reply data is adjusted and optimized to ensure that the reply content is more in line with the emotional state of the target object at that time, so as to provide a personalized reply. After such adjustments, the target reply data for the specific questions and emotional state of the target object will eventually be generated, which solves the problem in the related technology that when the virtual person recommends products to the user, the intelligent virtual person may not notice the changes in the user's emotions, which may easily lead to poor user consultation experience and technical problems that affect the user's consultation experience. In this way, the emotional changes of the target object can be accurately captured to achieve more personalized and professional interactions and improve the effect of the user's consultation experience.

[0133] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0134] Based on the above embodiments and optional embodiments, an optional implementation is provided, which is described in detail below.

[0135] In the related technology, when a virtual person recommends a product to a user, the intelligent virtual person may not notice the changes in the user's emotions, which may easily lead to a poor user consultation experience and technical problems that affect the user consultation experience. No effective solution has been proposed yet.

[0136] To address the above-mentioned problems, no effective solution has been proposed yet.

[0137] In view of this, an optional implementation of the present invention provides a consultation response method, which can also be called a virtual human intelligent dialogue method, which can effectively solve the technical problem in the related technology that when a virtual human recommends a product to a user, the intelligent virtual human may not notice the user's emotional changes, which may easily lead to a poor user consultation experience and affect the user consultation experience.

[0138] S1, obtaining audio data of the target object's consultation;

[0139] For example, receiving the user's consultation data (same as the above-mentioned audio data), the consultation data can be voice, text, etc., which is not limited here;

[0140] S2, determining consulting question data, N target image frames, and target voice data based on the audio data, wherein the N target image frames are image frames whose key point variation index of the target object is greater than a first threshold, and the target voice data are data whose tone variation index of the target object is greater than a second threshold, and N is a positive integer greater than 1;

[0141] S3, determining first sub-emotion change data according to the N target image frames, and determining second sub-emotion change data according to the target voice data;

[0142] Specifically, determining the first sub-emotion change data according to the N target image frames in S3 further includes:

[0143] S311, determining M pieces of overall contour data corresponding to the overall contour of the target object and Q pieces of local contour data corresponding to the local contour of the target object in N target image frames, wherein M is a positive integer greater than 1, and Q is a positive integer greater than 1;

[0144] S312, determining a first contour offset corresponding to the overall contour based on the M overall contour data, and determining a second contour offset corresponding to the local contour based on the Q local contour data;

[0145] S313, determining an offset difference between the second contour offset and the first contour offset;

[0146] S314: Determine the first sub-emotion change data according to the offset difference.

[0147] Specifically, determining the second sub-emotion change data according to the target speech data in S3 also includes:

[0148] S321, determining speech parameters in the target speech data, wherein the speech parameters at least include: volume parameters and pitch parameters;

[0149] S322, determining volume change information according to the volume parameter, and determining pitch change information according to the pitch parameter;

[0150] S323, determining second sub-emotion change data according to the volume change information and the tone change information.

[0151] S4, determining the emotion change data of the target object according to the first sub-emotion change data and the second sub-emotion change data;

[0152] For example, the user's reply data is received and the user's emotions (the same as the above-mentioned emotion change data) are judged. When judging the user's emotions, a comprehensive judgment can be made based on the user's facial expressions, thinking time, tone, etc.

[0153] S5, determining initial response data corresponding to the consultation question data;

[0154] For example, based on the user's consultation data, a reply corresponding to the consultation, i.e., a reply text (same as the initial reply data mentioned above) is determined;

[0155] Specifically, S5 also includes:

[0156] S51, determining the response type to the consultation question data;

[0157] S52, when the answer type is a selection type, obtaining the preference behavior data of the target object, wherein the correlation index between the preference behavior data and the consultation question data is greater than a third threshold;

[0158] S53, determining initial response data based on the preference behavior data.

[0159] S6, adjusting the initial response data according to the emotion change data to obtain the target response data.

[0160] Specifically, S6 also includes:

[0161] S611, determining the target object's preference response data;

[0162] S612, adjusting the initial response data based on the preference response data and the emotion change data to obtain the target response data.

[0163] Specifically, S6 also includes:

[0164] S621, determining the language data of the target object in the audio data, wherein the language data includes sentence data and word data;

[0165] S622, determining target response data based on the language data;

[0166] S623, adjusting the initial response data according to the target response data and the emotion change data to obtain the target response data.

[0167] For example, based on the user's emotions (the same as the above-mentioned emotion change data), the reply tone and reply words (the same as the above-mentioned target reply data) are determined;

[0168] It should be noted that, if the user's identity information can be obtained, the user's identity and historical behavior (same as the above-mentioned preference behavior data) can be combined to recommend suitable products to the user and introduce the products based on his or her preferences. This can be determined by combining the micro-expression library and the emotional language analysis model.

[0169] Combine the reply text with the reply tone and wording, and reply to the customer again. Judge the user's acceptance of the reply based on their behavior and facial expression. If the acceptance level is low, decide whether to change the wording or replace the product to avoid negative emotions in the user.

[0170] Specifically, step S6 further includes step S7, wherein S7 includes:

[0171] S71, generating virtual human behavior parameters based on the audio data;

[0172] S72, driving the virtual human to perform actions according to the virtual human behavior parameters, and replying to the target response data.

[0173] For example, before replying to the user, it is also necessary to obtain multimodal data, which includes data from the surrounding environment and multimodal input data for interaction with the user, and match the multimodal output data with the execution parameters of the virtual person's mouth shape, facial expression, head movement and body movement (the same as the above-mentioned virtual person behavior parameters), and present the execution parameters in the preset display area. In the above-mentioned interaction process, if voice is received, the voice can be converted into text first, and the text question can be input into the language model corresponding to the scene to obtain the text answer, and the text answer can be converted into a voice answer, which is broadcast by the virtual digital person in combination with emotions and words.

[0174] Through the above optional implementation, at least the following beneficial effects can be achieved:

[0175] (1) Compared with the related art, the present invention obtains the audio data of the target object when consulting, and extracts the consulting question data, N target image frames, and target voice data from the audio data. Then, for the N target image frames, the first sub-emotion change data is determined, and for the target voice data, the second sub-emotion change data is determined, and the first sub-emotion change data and the second sub-emotion change data are combined to obtain the overall emotion change data of the target object. At the same time, according to the initial reply data corresponding to the consulting question data. Finally, according to the determined emotion change data, the initial reply data is adjusted and optimized to ensure that the reply content is more in line with the emotional state of the target object at that time, thereby providing a personalized reply. After such adjustment, the target reply data for the specific question and emotional state of the target object will eventually be generated, which solves the technical problem that when the virtual person recommends products to the user in the related art, the intelligent virtual person may not notice the emotional changes of the user, which may easily lead to poor user consultation experience and affect the user consultation experience. Thus, the emotional changes of the target object are accurately captured to achieve more personalized and professional interaction and improve the effect of user consultation experience.

[0176] (2) Compared with the related art, the present invention determines the offset difference based on the first contour offset and the second contour offset, thereby avoiding the influence of the overall change of a specific part (such as the sudden movement of the face to the left) on the virtual human's analysis of the local feature changes in the specific part of the target object. Therefore, it can be determined that the local feature changes in the specific part are caused by the emotional changes of the target object rather than the overall changes of the specific part. Therefore, the emotional changes of the target object can be captured more accurately, which helps to provide more personalized and professional interactions and improve the user consultation experience.

[0177] (3) Compared with the related art, the present invention recognizes and determines the second sub-emotion change data of the target object by integrating the volume change information and the tone change information, avoiding the limitation of relying solely on facial expressions or body language to recognize emotions. By deeply analyzing the slight changes in the voice, the virtual human can capture the deeper information of the target object's emotions. In particular, when the target object's face is partially blocked or the body posture changes are not obvious, voice analysis can serve as an important supplementary means of emotion recognition, thereby improving the accuracy and comprehensiveness of emotion recognition and enhancing the user consultation experience.

[0178] (4) Compared with related technologies, the present invention can determine the preferred response data of the target object and adopt the communication method preferred by the user to provide personalized responses based on the content, style or tone preferred by the target object, so that the virtual person can not only understand the user's needs, but also respond in a way that is closer to the user's emotions and communication preferences, thereby reducing interaction barriers, improving dialogue efficiency, and enhancing the user experience.

[0179] (5) Compared with the related art, the present invention can more accurately understand the target object's language expression habits by identifying and extracting the language data used by the target object from the audio data, including sentence data and word data, and then determine the target response data that conforms to its language characteristics. The initial response data is adjusted through the target response data and the emotion change data. The target response data obtained can not only meet the target object's response needs, but also resonate with the target object in terms of language style and emotional state, thereby helping to achieve smoother and more efficient communication, and thus helping to enhance the user consultation experience.

[0180] Example 2

[0181] The embodiment of the present application also provides a consultation reply device. It should be noted that the consultation reply device of the embodiment of the present application can be used to execute the consultation reply method provided by the embodiment of the present application. The consultation reply device provided by the embodiment of the present application is introduced below.

[0182] According to an embodiment of the present application, a device for implementing the above-mentioned consultation reply method is also provided. Figure 3 is a schematic diagram of a consultation reply device provided according to an embodiment of the present application, such as Figure 3 As shown, the device includes: an acquisition module 302, which is used to acquire audio data of the target object for consultation; a first determination module 304, which is connected to the acquisition module 302, and is used to determine the consultation question data, N target image frames, and target voice data based on the audio data, wherein the N target image frames are image frames whose part key point change index of the target object is greater than a first threshold, and the target voice data is data whose tone change index of the target object is greater than a second threshold, and N is a positive integer greater than 1; a second determination module 306, which is connected to the first determination module 304, and is used to determine the consultation question data, N target image frames, and target voice data based on the audio data, wherein the N target image frames are image frames whose part key point change index of the target object is greater than a first threshold, and the target voice data is data whose tone change index of the target object is greater than a second threshold, and N is a positive integer greater than 1. The image frame determines the first sub-emotion change data, and determines the second sub-emotion change data based on the target speech data; the third determination module 308 is connected to the above-mentioned second determination module 306, and is used to determine the emotion change data of the target object based on the first sub-emotion change data and the second sub-emotion change data; the fourth determination module 310 is connected to the above-mentioned third determination module 308, and is used to determine the initial reply data corresponding to the consultation question data; the adjustment module 312 is connected to the above-mentioned fourth determination module 310, and is used to adjust the initial reply data based on the emotion change data to obtain the target reply data.

[0183] Optionally, the device provided in the embodiment of the present application also includes an offset determination module, which is used to determine M overall contour data corresponding to the overall contour of the target object and Q local contour data corresponding to the local contour of the target object in N target image frames, wherein M is a positive integer greater than 1, and Q is a positive integer greater than 1; based on the M overall contour data, determine the first contour offset corresponding to the overall contour, and based on the Q local contour data, determine the second contour offset corresponding to the local contour; determine the offset difference between the second contour offset and the first contour offset; and based on the offset difference, determine the first sub-emotion change data.

[0184] Optionally, the device provided in the embodiment of the present application also includes a speech determination module for determining speech parameters in the target speech data, wherein the speech parameters include at least: volume parameters and pitch parameters; determining volume change information based on the volume parameters, and determining pitch change information based on the pitch parameters; determining second sub-emotion change data based on the volume change information and the pitch change information.

[0185] Optionally, the device provided in the embodiment of the present application also includes a preference determination module for determining the preference reply data of the target object; adjusting the initial reply data based on the preference reply data and the emotion change data to obtain the target reply data.

[0186] Optionally, the device provided in the embodiment of the present application also includes a language determination module, which is used to determine the language data of the target object in the audio data, wherein the language data includes sentence data and vocabulary data; based on the language data, target reply data is determined; based on the target reply data and emotion change data, the initial reply data is adjusted to obtain the target reply data.

[0187] Optionally, the device provided in the embodiment of the present application also includes a reply type determination module for determining the reply type of the consultation question data; when the reply type is a selection type, obtaining the preference behavior data of the target object, wherein the correlation index between the preference behavior data and the consultation question data is greater than a third threshold; and determining the initial reply data based on the preference behavior data.

[0188] Optionally, the device provided in the embodiment of the present application also includes a behavior determination module, which is used to generate virtual human behavior parameters based on audio data; drive the virtual human to perform actions based on the virtual human behavior parameters, and reply to target response data.

[0189] The consultation reply device provided in the embodiment of the present application solves the technical problem that when a virtual person recommends a product to a user in the related art, the intelligent virtual person may not notice the emotional changes of the user, which may easily lead to a poor user consultation experience and affect the user consultation experience. This achieves accurate capture of the emotional changes of the target object, so as to achieve more personalized and professional interaction and improve the effect of the user consultation experience.

[0190] It should be noted that the acquisition module 302, the first determination module 304, the second determination module 306, the third determination module 308, the fourth determination module 310, and the adjustment module 312 correspond to steps S202 to S212 in Example 1, and each module and the corresponding step implement the same instance and application scenario, but are not limited to the content disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units can be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above-mentioned modules can also be part of the device and can be run in the computer terminal 10 provided in Example 1.

[0191] Example 3

[0192] An embodiment of the present application may provide an electronic device, Figure 4 is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 4 As shown, the electronic device may include: one or more ( Figure 4 Only one is shown) a processor, a memory, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0193] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0194] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain audio data of the target object for consultation; determine the consultation question data, N target image frames, and target voice data based on the audio data, wherein the N target image frames are image frames whose part key point change index of the target object is greater than a first threshold, and the target voice data is data whose tone change index of the target object is greater than a second threshold, and N is a positive integer greater than 1; determine the first sub-emotion change data based on the N target image frames, and determine the second sub-emotion change data based on the target voice data; determine the emotion change data of the target object based on the first sub-emotion change data and the second sub-emotion change data; determine the initial reply data corresponding to the consultation question data; adjust the initial reply data based on the emotion change data to obtain the target reply data.

[0195] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: determine M overall contour data corresponding to the overall contour of the target object and Q local contour data corresponding to the local contour of the target object in N target image frames, where M is a positive integer greater than 1, and Q is a positive integer greater than 1; determine the first contour offset corresponding to the overall contour based on the M overall contour data, and determine the second contour offset corresponding to the local contour based on the Q local contour data; determine the offset difference between the second contour offset and the first contour offset; and determine the first sub-emotion change data based on the offset difference.

[0196] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: determine the voice parameters in the target voice data, wherein the voice parameters include at least: volume parameters and pitch parameters; determine the volume change information based on the volume parameters, and determine the pitch change information based on the pitch parameters; determine the second sub-emotion change data based on the volume change information and the pitch change information.

[0197] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: determine the preferred reply data of the target object; adjust the initial reply data based on the preferred reply data and the emotion change data to obtain the target reply data.

[0198] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: determine the language data of the target object in the audio data, wherein the language data includes sentence data and word data; determine the target reply data based on the language data; adjust the initial reply data based on the target reply data and emotion change data to obtain the target reply data.

[0199] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: determine the response type to the consultation question data; when the response type is a selection type, obtain the preference behavior data of the target object, wherein the correlation index between the preference behavior data and the consultation question data is greater than a third threshold; determine the initial response data based on the preference behavior data.

[0200] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: generate virtual human behavior parameters based on the audio data; drive the virtual human to perform actions based on the virtual human behavior parameters, and reply to the target response data.

[0201] According to an embodiment of the present application, an electronic device is provided. Figure 4 Only one is shown in the figure) processor, memory, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display, and the processor can also call the information and application stored in the memory through the transmission device to execute steps S202 to S212 in Example 1, which solves the technical problem that when the virtual person recommends products to the user in the related technology, the intelligent virtual person may not notice the emotional changes of the user, which may easily lead to poor user consultation experience and affect the user consultation experience. And then the emotional changes of the target object are accurately captured to achieve more personalized and professional interaction and improve the effect of user consultation experience.

[0202] It can be understood by those skilled in the art that Figure 4 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Figure 4 The structure of the electronic device is not limited. Figure 4 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 4 Different configurations are shown.

[0203] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0204] Example 4

[0205] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the inquiry reply method provided in the first embodiment.

[0206] Optionally, in this embodiment, it may be a computer-readable storage medium, the computer-readable storage medium including a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned consultation reply methods.

[0207] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0208] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the program steps of the consultation reply method.

[0209] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0210] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0211] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0212] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0213] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0214] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0215] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A consultation reply method, characterized in that: include: Acquire audio data of the target object's consultation; Determine consulting question data, N target image frames, and target voice data based on the audio data, wherein the N target image frames are image frames whose part key point change index of the target object is greater than a first threshold, and the target voice data is data whose tone change index of the target object is greater than a second threshold, and N is a positive integer greater than 1; Determine first sub-emotion change data according to the N target image frames, and determine second sub-emotion change data according to the target voice data; Determining emotion change data of the target object according to the first sub-emotion change data and the second sub-emotion change data; Determining initial response data corresponding to the consultation question data; The initial response data is adjusted according to the emotion transformation data to obtain target response data.

2. The method according to claim 1, characterized in that Determining first sub-emotion change data according to the N target image frames includes: Determine M pieces of overall contour data corresponding to the overall contour of the target object and Q pieces of local contour data corresponding to the local contour of the target object in N target image frames, wherein M is a positive integer greater than 1, and Q is a positive integer greater than 1; Determine a first contour offset corresponding to the overall contour according to the M overall contour data, and determine a second contour offset corresponding to the local contour according to the Q local contour data; determining an offset difference between the second contour offset and the first contour offset; The first sub-emotion change data is determined according to the offset difference.

3. The method according to claim 1, characterized in that: Determining second sub-emotion change data according to the target speech data includes: Determining speech parameters in the target speech data, wherein the speech parameters at least include: volume parameters and pitch parameters; Determining volume change information based on the volume parameter, and determining pitch change information based on the pitch parameter; Second sub-emotion change data is determined according to the volume change information and the tone change information.

4. The method according to claim 1, characterized in that: According to the emotion transformation data, the initial response data is adjusted to obtain target response data, including: Determining the preferred response data of the target object; The initial response data is adjusted according to the preference response data and the emotion change data to obtain the target response data.

5. The method according to claim 1, characterized in that According to the emotion transformation data, the initial response data is adjusted to obtain target response data, including: Determining language data of the target object in the audio data, wherein the language data includes sentence data and word data; determining target response data based on the language data; The initial response data is adjusted according to the target response data and the emotion change data to obtain the target response data.

6. The method according to claim 1, characterized in that Determining initial response data corresponding to the consultation question data includes: Determine the type of response to the consultation question data; In the case where the answer type is a selection type, obtaining the preference behavior data of the target object, wherein a correlation index between the preference behavior data and the consultation question data is greater than a third threshold; The initial response data is determined based on the preference behavior data.

7. The method according to any one of claims 1 to 6, characterized in that After adjusting the initial response data according to the emotion transformation data to obtain the target response data, the method further includes: Generating virtual human behavior parameters based on the audio data; The virtual human is driven to perform actions according to the virtual human behavior parameters, and the target response data is replied.

8. A consultation reply device, characterized in that: include: An acquisition module is used to acquire audio data of consultation conducted by a target object; A first determination module is used to determine consulting question data, N target image frames, and target voice data based on the audio data, wherein the N target image frames are image frames whose part key point change index of the target object is greater than a first threshold, and the target voice data is data whose tone change index of the target object is greater than a second threshold, and N is a positive integer greater than 1; A second determination module, configured to determine first sub-emotion change data based on the N target image frames, and to determine second sub-emotion change data based on the target voice data; A third determination module, configured to determine the emotion change data of the target object according to the first sub-emotion change data and the second sub-emotion change data; A fourth determination module, used to determine initial answer data corresponding to the consulting question data; The adjustment module is used to adjust the initial response data according to the emotion transformation data to obtain target response data.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is run, the device where the computer-readable storage medium is located is controlled to execute the consultation reply method described in any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A memory storing an executable program; A processor is used to run the program, wherein the program, when running, executes the consultation reply method described in any one of claims 1 to 7.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the consultation reply method described in any one of claims 1 to 7 are implemented.

Images