Human-computer interaction method and electronic equipment

Abstract

The invention provides a man-machine interaction method and electronic equipment. The method comprises the steps that the electronic equipment obtains active interaction information actively input by a user, and in the process that the user inputs the active interaction information, non-active interaction information non-actively expressed by the user is collected; the electronic equipment can obtain a corresponding first emotional feature set and a corresponding second emotional feature set based on the active interaction information and the non-active interaction information respectively, and further determines the emotion of the user based on the two feature sets. The electronic equipment can determine the emotion which the digital person needs to express based on the emotion of the user, so that the digital person expresses the emotion which can meet the emotion demand of the user.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence, and in particular to a human-computer interaction method and electronic device. Background Art

[0002] As digital human technology matures, the interaction between users and digital humans has achieved the transmission of multiple information based on text, voice, expression, gesture, etc., which makes the interaction between users and terminals closer to the interaction between people. This human-like interaction method has brought about new interactive experience requirements, namely emotional experience.

[0003] Service-oriented digital humans refer to intelligently driven digital humans that can provide users with guidance services, consulting services, and business services in the service field (such as financial services, government services, retail services, etc.). Users interact with digital humans based on their actual needs, and digital humans can provide users with the required services based on the content of the interaction with users. Summary of the invention

[0004] The present application provides a human-computer interaction method and an electronic device. In the method, the electronic device can express emotions through a digital human to meet the emotional needs of the user in a scene.

[0005] In the first aspect, the present application provides a human-computer interaction method. The method is applied to an electronic device, and the method includes: the electronic device obtains active interaction information input by a user, wherein the active interaction information is used to represent the user's needs in a target scene. Based on the active interaction information that has been obtained, the electronic device queries a first set of emotional features corresponding to the active interaction information in an experience knowledge base. Then, in the process of the user inputting active interaction information, the electronic device collects the user's non-active interaction information, and the non-active interaction information is used to characterize the user's user portrait. And, the electronic device determines a second set of emotional features based on the collected non-active interaction information. The electronic device determines the user's emotions in the target scene based on the first set of emotional features and the second set of emotional features. And, the electronic device determines the emotional expression features of a digital person based on the user's emotions in the target scene, and the emotional expression features include at least one of facial expressions, body movements, and voice. In this way, the electronic device combines the first set of emotional features and the second set of emotional features to determine the user's emotions, which can effectively improve the accuracy of user emotion recognition, thereby making the emotional expression of the digital person more in line with the user's current emotional needs.

[0006] For example, the user can speak to the digital person to input the corresponding active interaction information. The user can also input text through a virtual keyboard or a physical keyboard to input the corresponding active interaction information. In this application, the user's input method can be contact or non-contact, which is not limited in this application.

[0007] Exemplarily, the target scene may be a service hall scene such as a financial hall, or any other scene, and this application is not limited thereto.

[0008] Exemplarily, the user's demand refers to a specific demand in a target scenario. For example, in a financial hall, the active interaction information is "I lost my credit card", and the corresponding user demand is "report the loss of credit card".

[0009] Exemplarily, the non-active interaction information may be images and voices, where the images include the collected facial and body images of the user. Optionally, the non-active interaction information may also be understood as facial features, body features, and voice prosody obtained after further analysis of the images and voices.

[0010] Exemplarily, the electronic device may collect non-active interaction information through a camera and a microphone.

[0011] Exemplarily, a digital human is a virtual image displayed on a display screen of an electronic device, which may be a human image or other images, and this application does not limit this.

[0012] Exemplarily, the first emotion feature set may be a priori emotion feature set in the embodiment of the present application, and the second emotion feature set may be a posteriori emotion feature set in the embodiment of the present application.

[0013] Exemplarily, the electronic device may be any electronic device with a display screen, such as a stand screen, a television, a mobile phone, a tablet, a wearable device, a smart home device, and a car device.

[0014] In a possible implementation, after obtaining the active interaction information input by the user, the method further includes: the electronic device determines the user's user needs based on the active interaction information. The electronic device determines the target solution based on the user needs. Correspondingly, after determining the emotional expression characteristics of the digital human based on the first emotional feature set and the second emotional feature set, the method further includes: the electronic device generates a target animation of the digital human based on the target solution and the emotional expression characteristics. The electronic device plays the target animation. In this way, the digital human can provide specific solutions according to the user's needs. Moreover, in the process of providing solutions, the digital human is given an emotional expression method that is more in line with the user's emotional needs. While meeting business needs, the emotional needs of users are taken care of.

[0015] In one possible implementation, a target animation of a digital human is generated based on a target solution and emotional expression characteristics, including: an electronic device obtains scene environment characteristics in a target scene, and / or obtains user characteristics of a user based on non-active interaction information, wherein the user characteristics belong to a user portrait. Based on the scene environment characteristics and / or user characteristics, a display method of the animation part corresponding to the target solution is determined. The electronic device generates a target animation of a digital human based on the target solution, the display method corresponding to the target solution, and the emotional expression characteristics. In this way, the electronic device in the present application can be closer to the user's needs in the scene by combining the environment of the scene and / or the characteristics of the user, and actively provide the user with a more humane service based on the collected non-active interaction characteristics when the user has not expressed.

[0016] In a possible implementation, based on the target solution and the emotional expression characteristics, the target animation of the digital human is generated, including: the electronic device synthesizes the facial expression of the digital human based on the emotional expression characteristics. In this way, the electronic device in the present application can realize the emotional expression of the digital human by synthesizing the specific expression of the digital human, and can meet the emotional needs of the user while enriching the image and emotions of the digital human.

[0017] In a possible implementation, synthesizing the facial expression of a digital person based on the emotion expression feature includes: the electronic device determines the adjustment parameter of at least one facial unit of the digital person based on the emotion expression feature. The electronic device adjusts the at least one facial unit based on the adjustment parameter of the at least one facial unit. In this way, the electronic device adjusts the facial expression of the digital person by modularizing the facial features.

[0018] In a possible implementation, determining the user's emotions in a target scene based on a first emotion feature set and a second emotion feature set includes: the electronic device determines a target emotion feature set based on the first emotion feature set and the second emotion feature set; wherein the target emotion feature set includes at least one emotion feature, and the at least one emotion feature is used to represent the user's emotions when the user is in the target scene and inputs active interaction information. In this way, the electronic device can determine the user's emotions based on the prior emotion features and the a posteriori emotion features. Then, the emotions of the digital person are set to meet the user's emotional needs.

[0019] In one possible implementation, the target emotion feature set is the intersection of the first emotion feature set and the second emotion feature set; or, if the intersection of the first emotion feature set and the second emotion feature set is zero, and the first emotion feature set is not empty, then the target emotion feature set is the first emotion feature set; or, if the intersection of the first emotion feature set and the second emotion feature set is zero, and the first emotion feature set is empty, then the target emotion feature set is the second emotion feature set. In this way, the electronic device in the present application can obtain user emotions through any of the two emotion acquisition methods. Moreover, by taking the intersection of the two emotion feature sets to obtain the user's emotions, the correction of the second emotion feature set based on the first emotion feature set can be achieved to more accurately identify the user's emotions.

[0020] In a possible implementation, if the intersection of the first emotion feature set and the second emotion feature set is zero, and the first emotion feature set is not empty; or, if the intersection of the first emotion feature set and the second emotion feature set is zero, and the first emotion feature set is empty, the method further includes: if the second emotion feature set is not empty, then based on the second emotion feature set, updating the experience knowledge base. In this way, the electronic device can update the knowledge base as time and scenes change to meet the needs of different times and different scenes. Moreover, with the update of the knowledge base, the electronic device can more accurately identify the user's emotions.

[0021] In one possible implementation, based on active interaction information, a first set of emotion features corresponding to the active interaction information is queried in an experience knowledge base, including: the electronic device obtains a target event based on the active interaction information, and the target event matches the user's needs. Based on the target event, a first set of emotion features is queried in the experience knowledge base, and the experience knowledge base includes the correspondence between events and sets of emotion features. In this way, the electronic device can obtain the target event by identifying the active interaction information, and then determine the user needs and the corresponding set of prior emotion features (i.e., the first set of emotion features) based on the target event. The present application can obtain the corresponding user emotions through the active interaction information input by the user by pre-learning the correspondence between user emotions and corresponding user needs (or target events).

[0022] In a possible implementation, the active interaction information includes at least one of the following: voice information, text information, and gesture information. In this way, the electronic device can obtain the corresponding prior emotion characteristics based on the active interaction information such as voice information and text information input by the user. The electronic device can obtain the user's emotion characteristics in two different ways, respectively, to verify the obtained user emotion characteristics and improve the accuracy of user emotion characteristic recognition.

[0023] In a possible implementation, the non-active interaction information includes at least one of the following: user appearance feature information and voice rhythm feature information. In this way, the electronic device can obtain the corresponding a posteriori emotion feature based on the collected non-active interaction information non-actively expressed by the user.

[0024] In a second aspect, the present application provides an electronic device. The electronic device includes: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the electronic device performs the following steps: obtaining active interaction information input by the user, the active interaction information is used to represent the user's needs in the target scene; based on the active interaction information, querying the first emotion feature set corresponding to the active interaction information in the experience knowledge base; in the process of the user inputting active interaction information, collecting the user's non-active interaction information, the non-active interaction information is used to characterize the user portrait of the user; based on the non-active interaction information, determining the second emotion feature set; based on the first emotion feature set and the second emotion feature set, determining the user's emotions in the target scene; based on the user's emotions in the target scene, determining the emotional expression features of the digital person, the emotional expression features including at least one of facial expressions, body movements, and voice.

[0025] In one possible implementation, when the computer program is executed by one or more processors, the electronic device performs the following steps: determining the user's needs based on active interaction information; determining a target solution based on the user's needs; generating a target animation of a digital human based on the target solution and emotional expression characteristics; and playing the target animation.

[0026] In one possible implementation, when the computer program is executed by one or more processors, the electronic device performs the following steps: obtaining scene environment features in the target scene, and / or obtaining user features of the user based on non-active interaction information, the user features belonging to the user portrait; determining the display method of the animation part corresponding to the target solution based on the scene environment features and / or the user features; generating the target animation of the digital human based on the target solution, the display method corresponding to the target solution, and the emotional expression features.

[0027] In a possible implementation, when the computer program is executed by one or more processors, the electronic device executes the following steps: synthesizing the facial expression of a digital human based on the emotion expression features.

[0028] In one possible implementation, when the computer program is executed by one or more processors, the electronic device performs the following steps: determining adjustment parameters of at least one facial unit of the digital human based on the emotion expression characteristics; and adjusting at least one facial unit based on the adjustment parameters of the at least one facial unit.

[0029] In one possible implementation, when the computer program is executed by one or more processors, the electronic device performs the following steps: determining a target emotion feature set based on a first emotion feature set and a second emotion feature set; wherein the target emotion feature set includes at least one emotion feature, and the at least one emotion feature is used to represent the emotion of the user when the user is in a target scene and inputs active interaction information.

[0030] In one possible implementation, the target emotion feature set is the intersection of the first emotion feature set and the second emotion feature set; or, if the intersection of the first emotion feature set and the second emotion feature set is zero, and the first emotion feature set is not empty, then the target emotion feature set is the first emotion feature set; or, if the intersection of the first emotion feature set and the second emotion feature set is zero, and the first emotion feature set is empty, then the target emotion feature set is the second emotion feature set.

[0031] In one possible implementation, if the intersection of the first emotion feature set and the second emotion feature set is zero, and the first emotion feature set is not empty; or, if the intersection of the first emotion feature set and the second emotion feature set is zero, and the first emotion feature set is empty, the method also includes: if the second emotion feature set is not empty, updating the experience knowledge base based on the second emotion feature set.

[0032] In one possible implementation, when the computer program is executed by one or more processors, the electronic device performs the following steps: based on active interaction information, obtaining a target event, where the target event matches the user's needs; based on the target event, querying a first emotion feature set in an experience knowledge base, where the experience knowledge base includes a correspondence between events and emotion feature sets.

[0033] In a possible implementation manner, the active interaction information includes at least one of the following: voice information, text information, and gesture information.

[0034] In a possible implementation, the non-active interaction information includes at least one of the following: appearance feature information and speech prosody feature information of the user.

[0035] In a third aspect, the present application provides a human-computer interaction device, which includes: an active interaction information acquisition module, which is used to acquire active interaction information input by a user, and the active interaction information is used to represent the user's needs in a target scene. An emotional feature acquisition module, which is used to query a first emotional feature set corresponding to the active interaction information in an experience knowledge base based on the active interaction information. A non-active interaction information acquisition module, which is used to collect the user's non-active interaction information during the process of the user inputting active interaction information, and the non-active interaction information is used to characterize the user's user portrait. The emotional feature acquisition module is also used to determine a second emotional feature set based on the non-active interaction information. A first decision module, which is used to determine the user's emotions in the target scene based on the first emotional feature set and the second emotional feature set. A second decision module, which is used to determine the emotional expression characteristics of a digital person based on the user's emotions in the target scene, and the emotional expression characteristics include at least one of facial expressions, body movements, and voice.

[0036] In a possible implementation, the user demand acquisition module is further used to determine the user demand of the user based on the active interaction information. The first decision module is used to determine the target solution based on the user demand; the second decision module is used to generate the target animation of the digital human based on the target solution and the emotional expression characteristics. The display module is used to play the target animation.

[0037] In a possible implementation, the second decision module is used to: obtain scene environment features in the target scene, and / or obtain user features of the user based on non-active interaction information, the user features belonging to the user portrait; determine the display method of the animation part corresponding to the target solution based on the scene environment features and / or user features; generate the target animation of the digital human based on the target solution, the display method corresponding to the target solution and the emotional expression features.

[0038] In a possible implementation, the execution module is used to synthesize the facial expression of the digital human based on the emotion expression characteristics.

[0039] In a possible implementation, the execution module is further used to: determine an adjustment parameter of at least one facial unit of the digital human based on the emotion expression feature; and adjust the at least one facial unit based on the adjustment parameter of the at least one facial unit.

[0040] In one possible implementation, the first decision module is further used to: determine a target emotion feature set based on the first emotion feature set and the second emotion feature set; wherein the target emotion feature set includes at least one emotion feature, and the at least one emotion feature is used to represent the emotion of the user when he is in the target scene and inputs active interaction information.

[0041] In one possible implementation, the target emotion feature set is the intersection of the first emotion feature set and the second emotion feature set; or, if the intersection of the first emotion feature set and the second emotion feature set is zero, and the first emotion feature set is not empty, then the target emotion feature set is the first emotion feature set; or, if the intersection of the first emotion feature set and the second emotion feature set is zero, and the first emotion feature set is empty, then the target emotion feature set is the second emotion feature set.

[0042] In one possible implementation, if the intersection of the first emotion feature set and the second emotion feature set is zero, and the first emotion feature set is not empty; or, if the intersection of the first emotion feature set and the second emotion feature set is zero, and the first emotion feature set is empty, the emotion feature acquisition module is also used to update the experience knowledge base based on the second emotion feature set if the second emotion feature set is not empty.

[0043] In a possible implementation, the emotion feature acquisition module is used to: acquire a target event based on active interaction information, where the target event matches the user's needs; and query a first emotion feature set in an experience knowledge base based on the target event, where the experience knowledge base includes a correspondence between events and emotion feature sets.

[0044] In a possible implementation manner, the active interaction information includes at least one of the following: voice information, text information, and gesture information.

[0045] In a possible implementation, the non-active interaction information includes at least one of the following: appearance feature information and speech prosody feature information of the user.

[0046] In a fourth aspect, the present application provides a computer-readable medium for storing a computer program, wherein the computer program includes instructions for executing the method in the first aspect or any possible implementation of the first aspect.

[0047] In a fifth aspect, the present application provides a computer program comprising instructions for executing the method in the first aspect or any possible implementation of the first aspect.

[0048] In a sixth aspect, the present application provides a chip, the chip comprising a processing circuit and a transceiver pin, wherein the transceiver pin and the processing circuit communicate with each other through an internal connection path, and the processing circuit executes the method in the first aspect or any possible implementation of the first aspect to control the receiving pin to receive a signal and control the sending pin to send a signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a schematic diagram of the hardware structure of an electronic device shown as an example;

[0050] Figure 2 A schematic diagram of the software structure of an electronic device is shown as an example;

[0051] Figure 3 is a schematic diagram of an application scenario shown as an example;

[0052] Figure 4 A schematic diagram of an exemplary user interface;

[0053] Figure 5 The figure is a schematic diagram of a human-computer interaction method flow chart;

[0054] Figure 6 is a schematic diagram of module interaction shown as an example;

[0055] Figure 7 is a schematic diagram of an exemplary process of obtaining target emotion features;

[0056] Figure 8 is a schematic diagram of module interaction shown as an example;

[0057] Fig. 9 is a schematic diagram of module interaction shown as an example;

[0058] Fig.10 A schematic diagram of a digital human is shown as an example;

[0059] Fig.11 A schematic diagram of a digital human is shown as an example;

[0060] Fig.12 A schematic diagram of a digital human is shown as an example;

[0061] Fig.13 A schematic diagram of a digital human is shown as an example;

[0062] Fig.14 is a schematic diagram of an application scenario shown as an example;

[0063] Fig.15 Schematic diagram of the structure of the device shown as an example. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0065] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0066] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.

[0067] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0068] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0069] The human-computer interaction method in the embodiments of the present application can be applied to electronic devices such as computers, tablets, and vehicle-mounted devices, and can also be applied to other types of computer devices, which is not limited in the present application.

[0070] Figure 1 1 shows a schematic diagram of the structure of the electronic device 100. It should be understood that Figure 1 The illustrated electronic device 100 is merely one example of an electronic device, and the electronic device 100 may have more or fewer components than those shown in the figure, may combine two or more components, or may have a different configuration of components. Figure 1 The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0071] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0072] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0073] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0074] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0075] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0076] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0077] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0078] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0079] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0080] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0081] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0082] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that electronic device 100 can communicate with the network and other devices through wireless communication technology.

[0083] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0084] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0085] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0086] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0087] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0088] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0089] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0090] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0091] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0092] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0093] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0094] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0095] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0096] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0097] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0098] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0099] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0100] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

[0101] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0102] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover can be automatically unlocked.

[0103] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0104] The distance sensor 180F is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0105] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user holds the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0106] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0107] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0108] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of a processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0109] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0110] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180M can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M to realize a heart rate detection function.

[0111] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0112] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0113] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0114] Figure 2 For an exemplary diagram of the software structure of an electronic device, please refer to Figure 2, the electronic device includes but is not limited to: a long-term memory module, a working memory module, a perception module, a primary decision module, a secondary decision module, and an execution module. It should be noted that the embodiments of the present application only involve modules related to the implementation of the human-computer interaction method in the embodiments of the present application, and the electronic device may also include other applications, modules or services, which are not limited in the present application.

[0115] Exemplarily, the long-term memory module is used to maintain the experience knowledge base, and, based on the experience knowledge base, obtain active interaction information. Active interaction information is the information that the user consciously conveys to the digital human when the user actively communicates with the terminal (i.e., the digital human). Active interaction information may include, but is not limited to: text input by the user, voice or text information obtained by voice-to-text conversion, operation gestures, content selected by the user on the terminal device through gestures such as clicks (such as recommended questions, keywords, etc.), etc. Exemplarily, the long-term memory module maintains the experience knowledge base in a manner that the long-term memory module obtains the latest experience knowledge base from the server when it needs to obtain active interaction information. Optionally, the long-term memory module may also obtain the experience knowledge base from the server and save the experience knowledge base locally, and is responsible for updating the local experience knowledge base. Optionally, the long-term memory module may also periodically obtain the latest experience knowledge base from the server and save the obtained experience knowledge base locally.

[0116] The perception module is used to obtain non-active interaction information. Non-active interaction information is some signals or information transmitted unconsciously by the user, which can be understood as feature information used to characterize the user's profile. Non-active interaction information may include but is not limited to: appearance feature information and voice rhythm feature information. Among them, appearance feature information includes but is not limited to: facial features, body posture, etc.

[0117] The working memory module is used to obtain user emotions based on active interaction information and passive interaction information.

[0118] The primary decision-making module is used to determine the interaction goal and emotional goal based on user emotions and user needs. Among them, user needs are obtained based on active interaction information. For example, if the user needs to report the loss of a credit card and avoid losses, the interaction goal is to guide the user to complete the credit card loss reporting process. Optionally, the interaction goal can also be understood as a solution provided based on user needs. The emotional goal can be understood as being used to meet the user's current emotional needs. For example, if the user's emotional characteristic is sadness, the emotional goal of the digital human is to alleviate the user's sadness.

[0119] The secondary decision module is used to determine the implementation method of the interaction goal and the implementation method of the emotional goal based on user characteristics and interaction environment.

[0120] The execution module is used to call the task SDK (Software Development Kit) to complete the user's interaction goal according to the implementation method of the interaction goal and the implementation method of the emotion goal, so as to generate the animation of the digital human.

[0121] Figure 3 This is a schematic diagram of an exemplary application scenario, in which the application scenario includes but is not limited to: a user and a stand screen 30 (which may also be called a display screen or a large screen, which is not limited in this application). A digital human 301 is displayed on the stand screen 30 .

[0122] For example, the user can transmit active interaction information to the digital human 30 through text input, voice commands, screen clicks, gestures and other interactive methods. At the same time, the user's facial expressions, body postures and other non-active interaction information will also be collected by the digital human 301 (i.e., the terminal).

[0123] The hardware structure of the vertical screen 30 can be referred to Figure 1 That is, the stand screen 30 may include but is not limited to devices or modules such as cameras, touch screens, sensors, etc., which can be used to obtain information input by the user (i.e., active interaction information), and can also collect non-active interaction information. The processor in the stand screen 30 can be used to support the implementation of Figure 2 The implementation of each module in .

[0124] Optionally, the vertical screen 30 may also be a display screen of a terminal device, wherein a camera and a microphone (ie, a receiver) are provided on the display screen.

[0125] Figure 3 The application scenario shown is an offline interactive scenario. For example, the vertical screen can be set up in a service hall or other places. The method in the embodiment of the present application can also be applied to online interactive scenarios, that is, the digital human can be presented on mobile phones, tablets, computers, televisions, vehicle-mounted devices, wearable devices (such as smart watches, smart bracelets, etc.), speakers with screens and other devices. It should be noted that the digital human can be in the form of a portrait, a cartoon image, or an image of other animals or robots, etc., and this application does not limit this. In other words, in the embodiment of the present application, the terminal device can virtualize a digital human to achieve the purpose of communication and communication with the user through the digital human.

[0126] like Figure 4 As shown, a human-computer interaction application can be installed in the user's mobile phone. If the user clicks the human-computer interaction application icon, the mobile phone can display the human-computer interaction interface in response to the received user operation. The human-computer interaction interface may include but is not limited to a digital human display window, in which the digital human can be displayed. Optionally, the digital human displayed in the window may only include a partial screen of the digital human. For example, the whole body of the digital human can be displayed in the vertical screen, and the upper body of the digital human can be displayed in the window of the mobile phone, which is not limited in this application.

[0127] Optionally, the digital human in the embodiment of the present application can also be presented in a virtual space environment. Accordingly, the user referred to in the embodiment of the present application is usually a virtual image representing the user in a virtual real scene. The user performs interactive behaviors by controlling the virtual image, and the digital human in the virtual world uses the same logic to interact with the user.

[0128] The devices and modules (such as cameras, sensors, microphones, etc.) used to collect active interaction information and passive interaction information and the display screen used to display the digital human in the embodiments of the present application can be collectively referred to as external devices. Figure 2 The processor implementing each module in the embodiment may be arranged in the same electronic device, for example, Figure 3 The stand screen in the embodiment may include a processor and an external device, the external device is used to collect information and display the digital human, and the processor is used to generate an animation of the digital human based on the information collected by the external device and display it on the display screen. In another possible implementation, the external device and the processor may not be in the same device. For example, the external device may be a mobile phone or a stand screen, and Figure 2 That is, in this example, the mobile phone or the vertical screen can obtain active interaction information and non-active interaction information through devices such as cameras and microphones, and send the active interaction information and non-active interaction information to the cloud server. Figure 2 Each module in the system can generate a digital human animation and send the digital human animation to a mobile phone, and the display screen of the mobile phone can display the digital human animation.

[0129] The embodiment of the present application will be combined with a standing screen set up in a financial service hall to explain the human-computer interaction method in the embodiment of the present application in detail. Figure 5 For an exemplary human-computer interaction method flow chart, please refer to Figure 5 , including but not limited to the following steps:

[0130] S501, the electronic device receives active interaction information input by a user, and collects inactive interaction information of the user.

[0131] For example, Figure 2 In the scene shown, the stand screen is set in the financial service hall, and the stand screen can display the animation of the digital human, which can be called the first preset animation. In the first preset animation, the digital human can stand still or make some simple movements, such as greeting, etc., which is not limited in this application.

[0132] For example, the camera and infrared sensor of the stand-up screen can collect information in real time, and determine whether there is a user standing in front of the stand-up screen based on the collected information, and the distance between the user and the stand-up screen is a preset distance (for example, 1 meter, which can be set according to actual needs and is not limited in this application). The specific identification method can refer to the existing technology and is not limited in this application.

[0133] When the stand screen determines that a user is standing in front of the stand screen, and the distance between the user and the stand screen is the preset distance, the stand screen can start to execute the process of S501, that is, the stand screen starts the microphone and starts to collect information through the camera, microphone and other devices. In addition, the stand screen can play the second preset animation. In the second preset animation, the digital human can make a voice "Hello, how can I help you?" and make a listening gesture, so as to prompt the user to interact with the digital human to obtain the required help. The specific animation can be set according to actual needs and is not limited in this application.

[0134] For example, the user can express the user needs in the current scenario through button operation, gesture operation, voice input, text input, etc.

[0135] Taking the scenario of the financial service hall as an example, suppose that in the current scenario of the service hall, user 1 is a middle-aged woman around 55 years old. The user has lost her credit card and goes to the financial service hall to reissue the credit card. User 1 comes to the digital person (i.e. the standing screen) and says to the digital person, "I lost my credit card."

[0136] In this example, the vertical screen can collect the user's voice through the microphone, that is, obtain the audio of the user saying "I lost my credit card". In addition, the camera of the vertical screen collects the user's facial image at the same time. In other words, the vertical screen not only obtains the user's input voice "I lost my credit card", but also collects the corresponding facial image when the user inputs the voice.

[0137] Exemplarily, Liping can convert the acquired voice signal (also called audio signal) into text information, which is the active interaction information of the user.

[0138] It should be noted that in the embodiment of the present application, the camera of the vertical screen collects data in real time, that is, the images acquired by the vertical screen include images during, before, and after the conversation between the user and the robot object. For example, the audio and images acquired by the vertical screen may include time tags, and the vertical screen can determine the corresponding acquisition time of the audio and image based on the time tags of the audio and image. In this way, the vertical screen can determine which image frames are acquired during the process of the user inputting automatic interaction information.

[0139] For example, the vertical screen can perform image recognition on the user's facial image based on the image captured by the camera to obtain information such as the user's facial features, age range, gender, etc. In addition, the vertical screen can also analyze the audio information collected by the microphone to obtain the rhythmic features in the speech. For example, the rhythmic features of the speech include but are not limited to: fundamental frequency, duration and energy.

[0140] Among them, fundamental frequency: sound is composed of many sine waves of different frequencies, among which the lowest frequency sine wave is the fundamental frequency, and the frequency of the fundamental frequency is the fundamental frequency, which reflects the periodic vibration of the vocal cords when making voiced sounds. In language, the changing pattern of the fundamental frequency is called tone, and the tone (pitch of voice) contains important emotional information. In music and singing, the fundamental frequency is the main element to distinguish the pitch.

[0141] Timing: refers to the speed of speaking, which reflects the characteristics of speed of voice. The speaker's speaking speed will also change under different emotional states. Generally speaking, when the speaker is excited, the speaking speed is faster than the normal speaking speed.

[0142] Power: refers to the amplitude characteristics of speech signals, reflecting the loudness of voice. For emotions with high arousal, such as anger, happiness, fear, surprise, etc., the amplitude of speech signals is often large; while for emotions with low arousal, such as sadness, the amplitude of speech signals is often small.

[0143] In the embodiment of the present application, facial features, age range, gender, posture and rhythm features can be referred to as non-active interaction information. In other words, non-active interaction information is some signals transmitted unconsciously by the user, and non-active interaction features can be used to characterize user portraits.

[0144] S502, the electronic device determines prior emotion characteristics and user needs based on the active interaction information and the experience knowledge base.

[0145] In an embodiment of the present application, the cloud server may store an experience knowledge base, and the electronic device may obtain the experience knowledge base from the cloud server through a communication connection with the cloud server.

[0146] Exemplarily, when executing S502, the long-term memory module in the electronic device obtains the latest experience knowledge base from the cloud server. Optionally, after the long-term memory module obtains the experience knowledge base from the server, it saves the experience knowledge base locally and is responsible for updating the local experience knowledge base (the specific update process will be described below). Optionally, the long-term memory module can also periodically obtain the latest experience knowledge base from the cloud server and save the obtained experience knowledge base locally. The specific acquisition method can be set according to actual needs, and this application is not limited.

[0147] In an embodiment of the present application, the experience knowledge base records common scene events and their categories in the current application scenario, as well as the emotional characteristics corresponding to each category and the user needs corresponding to the scene events.

[0148] The following is a detailed description of the training process of the experience knowledge base:

[0149] The experience knowledge base can obtain a series of scene events in a specific scenario through interviews and field research (usually in a certain scenario, such as the financial service hall scenario, the number of scene events is limited and can be exhaustive). Then, through a large sample survey, users are asked to annotate the emotions under each event, and the events are clustered based on the annotated results. The results show that common service scene events include but are not limited to five categories, each of which has obvious emotional characteristics.

[0150] Exemplarily, the process of clustering events through annotation results is as follows:

[0151] Step 1: Through interviews and research, we find that there are n events S in the financial service scenario: {S1, S2, S3, …, S n}.

[0152] Step 2: A questionnaire survey is conducted on a large sample (the total number of users is represented by N), asking users to label their emotions for each event S, that is, to choose their own emotions when facing the event from the seven basic emotions {E1, E2, E3, E4, E5, E6, E7}.

[0153] Step 3: After the questionnaire survey is completed, calculate the selection ratio of each type of emotion E in each event S. The calculation method is as follows:

[0154] For event S j , if x users choose emotion E q , then event S j E q The sentiment ratios are:

[0155]

[0156] Then, event Sj There are 7 corresponding eigenvalues, namely the emotion ratio values ​​from E1 to E7: {P j1 ,P j2 ,P j3 ,P j4 ,P j5 ,P j6 ,P j7}.

[0157] Step 4: Perform cluster analysis based on the seven eigenvalues ​​of event S using hierarchical clustering methods. Specifically, use Euclidean distance to calculate the similarity between different events, and then group the events with the smallest distance into clusters. The clustering method uses average linkage.

[0158] Event S j With event S k The distance calculation formula is:

[0159]

[0160] Combine the two events S with the smallest distance D into a combined event (S j ,S k ), and then recalculate the distance matrix between each category event S. The distance calculation method between event points is the same as above. It should be noted that the combination (S j ,S k ) and S m The calculation method between events is to calculate S j To S m and S k To S m The distance is then averaged, that is:

[0161]

[0162] Event combination (S j ,S k ) and event combination (S m ,S n ) is calculated using the average connection method, that is:

[0163]

[0164] According to the above method, the distances between events, event combinations, and event combinations are repeatedly calculated, and the combinations with the smallest distances are combined to achieve hierarchical clustering of events.

[0165] After the cluster analysis is completed, the emotion ratio values ​​of each event in the same category are averaged, and the emotions with a total emotion ratio exceeding 50% are taken as the typical emotion characteristics of the category. That is, the corresponding relationship between scene categories and emotion characteristics is obtained.

[0166] Furthermore, the scenario can be summarized based on the common rules of different events in the same scenario. For example, the loss of credit card and debit card are both serious problem situation events caused by the user himself.

[0167] Taking the financial services scenario as an example, Table 1 is the result of clustering analysis of events in the financial services scenario based on emotional characteristics, which shows the five major categories of common scenario events in the financial services scenario and the emotional characteristics corresponding to each category.

[0168] Table 1

[0169]

[0170] For example, Figure 6 For an exemplary diagram of module interaction, please refer to Figure 6 , the long-term memory module can obtain prior emotion features (also known as prior emotion information) based on active interaction information and the experience knowledge base. Specifically, since experience-based processing is a faster and more efficient way of processing information, the long-term memory module gives priority to determining whether the current event is an event that the system is familiar with (that is, an event that has been recorded in the experience knowledge base). Exemplarily, the long-term memory module can perform semantic recognition on active interaction information to determine the corresponding event.

[0171] like Figure 6 As shown, in one example, if it is a familiar event (i.e., the event has been recorded in the experience knowledge base), the long-term memory module can search the experience knowledge base for the scene category corresponding to the event. Then, the long-term memory module can obtain the emotion features corresponding to the scene category from the experience knowledge base based on the acquired scene category, i.e., the prior emotion features. In the embodiment of the present application, the prior emotion features can also be referred to as a prior emotion feature set.

[0172] The long-term memory module outputs the acquired prior emotion feature set to the working memory module.

[0173] For example, still taking the example in the above text as an example, the long-term memory module obtains active interaction information, and the active interaction information includes text information corresponding to "I lost my credit card". The long-term memory module performs semantic recognition (also known as semantic analysis) on the text information corresponding to the active interaction information, and obtains the keyword "credit card lost", and can determine the corresponding event as "credit card lost" based on the keyword. Based on the "credit card lost" event, the long-term memory module queries the experience knowledge base (i.e., Table 1). In addition, the long-term memory module obtains the category corresponding to the "credit card lost" event as Category 1: serious problem situations caused by the user's own reasons. Based on Category 1, the long-term memory module further determines that the prior emotional features included in the user's prior emotion set are: sadness and panic.

[0174] Still refer to Figure 6 In another example, if the event is an unfamiliar event, that is, the long-term memory module does not find the event in the experience knowledge base. In one possible implementation, the long-term memory module can send a priori emotional feature set to the working memory module, and the priori emotional feature set is empty. Optionally, the long-term memory module may not feed back the priori emotional features to the working memory module. Optionally, the long-term memory module may also feed back information with other identifiers to the working memory module to indicate that the priori emotional features have not been obtained. In another possible implementation, the long-term memory module may identify the corresponding priori emotional features online by semantic recognition and other methods. Specifically, the long-term memory module may perform semantic recognition on the text information corresponding to the active interaction information to determine whether the user demand (ie, the event) is a problem situation. Exemplarily, if it is not a problem situation, the event can be classified as "consulting about financial products" and the corresponding category is determined to be category five: non-problem situation. And, the long-term memory module can obtain the corresponding priori emotional feature set, including prior emotional features such as neutrality and joy. Exemplarily, if it is a problem situation, semantic recognition can be performed on the text information corresponding to the active interaction information to determine whether the problem is caused by the user or others. In addition, the severity of the problem can be further identified. For example, a serious problem involves the risk of monetary loss, and a non-serious problem involves the risk of time loss. In this way, the long-term memory module can determine which category in Table 2 the unidentified event belongs to based on the cause of the problem and the severity of the problem. For example, if the long-term memory module determines that the problem is a serious problem caused by others based on the active interaction information, it can be determined to belong to category three. The long-term memory module can further obtain the prior emotional features corresponding to the category.

[0175] The long-term memory module outputs the acquired prior emotion feature set to the working memory module.

[0176] Optionally, if the long-term memory module still fails to identify the corresponding category based on the above judgment process, and the corresponding emotional features are not obtained. The long-term memory module can send a priori emotional feature set to the working memory module, and the prior emotional feature set is empty. Optionally, the long-term memory module may not feed back the prior emotional features to the working memory module. Optionally, the long-term memory module may also feed back information with other identifiers to the working memory module to indicate that the prior emotional features have not been obtained.

[0177] Exemplarily, the experience knowledge base may also include the correspondence between careers and businesses. Based on the identified event, the long-term memory module may obtain the business corresponding to the event from the experience knowledge base, thereby obtaining a solution that meets the user's needs. For example, in the aforementioned example, the user's credit card is lost. After the long-term memory module identifies the event as "credit card loss", it may query the experience knowledge base and obtain the business corresponding to the "credit card loss" event, which is reporting a lost credit card. In other words, based on the acquired event, the long-term memory module may determine that the user's need in the current scenario is to report a lost credit card.

[0178] For example, Figure 6 As shown, the long-term memory module outputs the acquired user demand (ie, the corresponding event, such as "credit card lost") to the working memory module.

[0179] S503, the electronic device determines a posteriori emotion feature based on the non-active interaction information.

[0180] For example, refer to Figure 6 , the perception module in the electronic device can obtain non-active interaction information. Specifically, the electronic device obtains the image captured by the camera and the voice collected by the microphone. The recognition module (such as NPU) in the electronic device can recognize the image captured by the camera to obtain the user's facial features and / or body posture and other features. In addition, the recognition module can also obtain the voice rhythm features based on the voice collected by the microphone.

[0181] It should be noted that some semantic recognition, image recognition, audio recognition and analysis technologies involved in the embodiments of this application can refer to any feasible method in the existing technology, and this application does not limit it.

[0182] The perception module can obtain information such as facial features, body postures, and speech rhythm features. The perception module can analyze facial features to determine user expressions, and based on the acquired user expressions, determine the user's emotions, that is, the delayed emotional features. (Related technologies can refer to existing face recognition technologies, which are not limited in this application). In addition, the perception module can determine user emotions based on body postures, that is, the posterior emotional features. And, the perception module can also determine the corresponding user emotions based on speech rhythm features, that is, the posterior emotional features. In this way, the perception module can obtain multiple posterior emotional features based on non-active interactive information and based on different modal recognitions, and the multiple posterior emotional features can be the same or different. For example, the perception module recognizes that the corresponding posterior emotional features are "sadness" based on facial features, and the corresponding posterior emotional features are "sadness" and "anger" based on speech rhythm. Correspondingly, the perception module can obtain a posterior emotional set, which includes multiple posterior emotional features recognized by different modalities.

[0183] like Figure 6 As shown, the perception module outputs the acquired posterior emotion features to the working memory module.

[0184] Exemplarily, the perception module can also obtain user features of a specified type based on non-active interaction information. For example, the perception module can obtain feature information such as the user's age and gender that can be used to describe the user's portrait based on facial features.

[0185] Still refer to Figure 6 ,The perception module outputs the acquired user features to the working memory module.

[0186] S504, the electronic device determines the target emotion feature based on the prior emotion feature and the a posteriori emotion feature

[0187] For example, Figure 6 As shown, the working memory module in the electronic device can determine the target emotion feature set based on the acquired prior emotion feature set and the acquired posterior emotion feature set. In the embodiment of the present application, the target emotion feature set includes at least one target emotion feature, which is used to indicate the emotion of the user in the current scene, and can be more accurately understood as the emotion expressed by the user when inputting active interaction information.

[0188] Specifically, the working memory module determines the target emotional characteristics based on the following rules:

[0189] 1. Obtain the intersection of the prior emotion feature set and the posterior emotion feature set. If the intersection is not empty, the target emotion feature set is the intersection.

[0190] 2. Obtain the intersection of the prior emotion feature set and the posterior emotion feature set. If the intersection is empty and the prior emotion feature set is not empty, the target emotion feature set is the prior emotion feature set.

[0191] 3. Obtain the intersection of the prior emotion feature set and the posterior emotion feature set. If the intersection is empty, and the prior emotion feature is empty, but the posterior emotion feature is not empty, then the target emotion feature set is the posterior emotion feature set.

[0192] like Figure 7 As shown in the figure, it is a schematic diagram of the process of obtaining target emotional features. Please refer to Figure 7 , including but not limited to:

[0193] S701, determining whether there is an intersection between the priori emotion feature set and the posteriori emotion feature set.

[0194] Exemplarily, after the working memory module obtains the prior emotion feature set and the posterior emotion feature set, it can compare the prior emotion feature set and the posterior emotion feature set to determine whether there is an intersection between the prior emotion feature set and the posterior emotion feature set.

[0195] In one example, if there is an intersection, S702 is executed.

[0196] In another example, if there is no intersection, S703 is executed.

[0197] S702: Determine that the target emotion set is an intersection.

[0198] In one example, if the prior emotion feature set is exactly the same as the posterior emotion feature set, that is, there is an intersection between the prior emotion feature set and the posterior emotion feature set, then the target emotion feature set is the intersection, that is, the prior feature emotion set or the posterior feature emotion set. For example, still taking the above example, after user 1 loses his credit card, it is assumed that the emotion feature identified by the perception module based on facial features is sadness, and the emotion feature identified based on the rhythmic features of the voice is panic, then the long-term memory module determines the posterior emotion feature set to be {sadness, panic}. And, the long-term memory module determines the prior emotion feature set to be {sadness, panic} based on active interaction information. The working memory module obtains the prior emotion feature set to be {sadness, panic} and the posterior emotion feature set to be {sadness, panic}. The working memory module detects that the prior emotion feature set {sadness, panic} is the same as the posterior emotion feature set {sadness, panic}, that is, the intersection of the two sets is: {sadness, panic}. The working memory module can determine the target emotion feature set, that is, the user's emotion feature is {sadness, panic}.

[0199] In another example, if the prior emotion feature set is not completely the same as the posterior emotion feature set, that is, there is an intersection between the prior emotion feature set and the posterior emotion feature set, then the intersection is the target emotion feature set. For example, still taking the above example, after user 1 loses his credit card, it is assumed that: the long-term memory module determines the prior emotion feature set as {sadness, panic} based on active interaction information, the emotion feature identified by the perception module based on facial features is sadness, and the emotion feature identified based on the rhythmic features of the voice is sadness, then the perception module determines the posterior emotion feature set as {sadness}. The working memory module obtains the prior emotion feature {sadness, panic} and the posterior emotion feature {sadness}, and the working memory module detects that the intersection of the prior emotion feature {sadness, panic} and the posterior emotion feature {sadness} is {sadness}, then the working memory module determines the target emotion feature set, that is, the user's emotion feature is {sadness}.

[0200] Let's take another example. Assume that: the long-term memory module determines the prior emotion feature set as {sadness, panic} based on active interaction information, the emotion feature identified by the perception module based on facial features is sadness, and the emotion feature identified based on the rhythmic features of the voice is surprise, then the perception module determines the posterior emotion feature set as {sadness, surprise}. The working memory module obtains the prior emotion feature {sadness, panic} and the posterior emotion feature {sadness, surprise}, and the working memory module detects that the intersection of the prior emotion feature {sadness, panic} and the posterior emotion feature {sadness, surprise} is {sadness}, then the working memory module determines the target emotion feature set, that is, the user's emotion feature is {sadness}.

[0201] S703, determining whether the prior emotion feature set is empty.

[0202] Please continue to refer to Figure 7 If the prior emotion feature set is completely different from the posterior emotion feature set, that is, the working memory module determines that the prior emotion feature set has no intersection with the posterior emotion feature set (that is, the intersection is empty), the working memory module detects whether the prior emotion feature set is empty.

[0203] In one example, if the prior emotion feature set is empty, execute S708.

[0204] In another example, if the prior emotion feature set is not empty, S704 is executed.

[0205] S704: Determine that the target emotion set is a priori emotion set.

[0206] Exemplarily, if the working memory module determines that the prior emotion feature set is not empty (ie, not null), the prior emotion feature set is determined to be the target emotion set, that is, the emotion feature of the user.

[0207] S705, determining whether the accumulated number of times reaches a threshold.

[0208] In this example, the working memory module detects the count list to be updated maintained by the long-term memory module (it may also be in other forms, which are not limited in this application) based on the event (e.g., "credit card lost") and the posterior emotion feature set. The count list to be updated includes the event, the corresponding posterior emotion feature set, and the accumulated number of times.

[0209] Exemplarily, the working memory module detects whether the event and the corresponding posterior emotion feature set are included in the count list to be updated. If so, it detects whether the corresponding cumulative count reaches a threshold.

[0210] In one example, if the accumulated number reaches a threshold, S707 is executed.

[0211] In another example, if the accumulated number of times does not reach the threshold, S706 is executed.

[0212] S706, accumulate the number of times.

[0213] Exemplarily, if the cumulative number of times does not reach the threshold, the working memory module sends cumulative indication information to the long-term memory module, and the cumulative indication information includes an event and a posterior emotion feature set, which is used to indicate the cumulative number of times corresponding to the event and the posterior emotion feature set. In response to the instruction of the working memory module, the long-term memory module searches for the table item corresponding to the event and the posterior emotion feature set in the count list to be updated, and accumulates the corresponding number of times once. If the table item is not included in the list, the table item can be added to the list and the number of times can be accumulated.

[0214] S707, update the experience knowledge base.

[0215] Exemplarily, if the cumulative number reaches a threshold (for example, 3, which can be set according to actual needs and is not limited in this application), the working memory module can send update indication information to the long-term memory module, and the update indication information includes an event and a posterior emotion feature set, which is used to indicate that the experience knowledge base is updated based on the event and the posterior emotion feature set. In response to the update indication information of the working memory module, the long-term memory module searches the experience knowledge base for whether the event already exists.

[0216] In one example, if the event already exists, the user emotion corresponding to the event may be updated, that is, the emotion features in the posterior emotion feature set are added to the existing user emotions.

[0217] In another example, if the event does not exist, a category corresponding to the event and the user emotion may be created in the experience knowledge base.

[0218] Optionally, if the posterior emotion feature set is also empty, this process ends.

[0219] For example, suppose that: the long-term memory module determines the prior emotion feature set as {sadness, panic} based on active interaction information, the emotion feature identified by the perception module based on facial features is anger, and the emotion feature identified based on the rhythmic features of the voice is surprise, then the perception module determines the posterior emotion feature set as {anger, surprise}. The working memory module obtains the prior emotion feature {sadness, panic} and the posterior emotion feature {anger, surprise}, and the working memory module detects that the intersection of the prior emotion feature {sadness, panic} and the posterior emotion feature {anger, surprise} is empty. The working memory module further determines that the prior emotion feature is not empty, and then it can be determined that the target emotion feature set is the prior emotion feature set, that is, {sadness, panic}. The working memory module detects the corresponding table items of the "credit card lost" event and the posterior emotion feature {anger, surprise} in the count list to be updated maintained by the long-term memory module, and the cumulative number of times in the table item is 3, and it can be determined that the cumulative number of times has reached the threshold (for example, the threshold is 3). The working memory module may send update indication information to the long-term memory module, and the update indication information includes the event "credit card lost" and the posterior emotion feature set {anger, surprise}. In response to the update indication information, the long-term memory module searches for the event "credit card lost" in the experience knowledge base and determines that relevant information corresponding to the event already exists. The long-term memory module may add the posterior emotion feature set {anger, surprise} to the user emotion item corresponding to the category corresponding to the event "credit card lost". Combined with Table 1, the updated experience knowledge base is shown in Table 2:

[0220] Table 2

[0221]

[0222] Optionally, as described above, the experience knowledge base may be maintained and updated by the cloud server. In this example, the long-term memory module may transmit the updated experience knowledge base to the cloud server. Optionally, the long-term memory module may also transmit the part that needs to be updated to the cloud server, which is not limited in this application.

[0223] S708, determining the target emotion set as the posterior emotion set.

[0224] Exemplarily, if the working memory module determines that the a priori emotion feature set is empty, the a posteriori emotion feature set is determined to be the target emotion set, that is, the emotion feature of the user.

[0225] The working memory module executes S705, S706 and S707. The details can be found above and will not be repeated here.

[0226] Thus, in the embodiment of the present application, the prior emotion features and the posterior emotion features are obtained respectively through two channels, and the user's emotions are determined based on the intersection of the two emotion feature sets, so as to realize the correction of the posterior emotion features by the prior emotion features, which can also be understood as the correction of the prior emotions based on the posterior emotion features. Moreover, since the prior emotion features and the posterior emotion features are obtained respectively through two channels in the embodiment of the present application, when the prior emotion features or the posterior emotion features are not successfully obtained, that is, when the corresponding set is empty, the user emotion features can be obtained based on the emotion features obtained by the other channel. This is more stable than judging the emotion features by a single event itself, and the experience knowledge base that the cloud server needs to store is more systematic.

[0227] In a possible implementation, as described above, active interaction information and non-active interaction information may correspond to a timestamp (i.e., a time tag) to indicate the acquisition time corresponding to the active interaction information and the non-active interaction information. In this way, after acquiring the prior emotion feature, the electronic device may acquire the non-active interaction information acquired in the corresponding time period based on the timestamp corresponding to the active interaction information, and acquire the corresponding a posteriori emotion feature based on the non-active interaction information in the time period. In other words, the electronic device may acquire the non-active interaction information in real time before, during, and after the user inputs the active interaction information, that is, collect information such as images and audio in real time, and perform the a posteriori emotion feature analysis process described above. For example, at time T1, the user has not yet input the active interaction information, and the electronic device may have acquired the non-active interaction information and acquired the corresponding a posteriori emotion feature. Optionally, the long-term memory module may send a corresponding timestamp while sending the a priori emotion feature. The timestamp is the timestamp corresponding to the active interaction information on which the a priori emotion feature is based. Similarly, when the perception module sends the a posteriori emotion feature to the working memory module, it can also send the corresponding timestamp, which is the timestamp corresponding to the non-active interaction information based on the a posteriori emotion feature. The working memory module can determine the target time period for the user to input active interaction information based on the timestamp corresponding to the a priori emotion feature. The working memory module can find the a posteriori emotion feature acquired within the target time period based on the timestamp corresponding to the a posteriori emotion feature.

[0228] S505, the electronic device determines the interaction target and the emotional target based on the user's needs and emotional characteristics

[0229] like Figure 8As shown, the primary decision module in the electronic device can obtain the user emotions and user needs input by the working memory module. Among them, the user emotions are the target emotion feature set obtained by the working memory module in S504, and the set includes at least one target emotion feature for representing the corresponding user emotions. Exemplarily, the user needs output by the working memory module are the events input by the long-term memory module to the working memory module, such as the "credit card lost" event.

[0230] In the embodiment of the present application, the working memory module can also pre-learn the user needs for different events. For example, the event input by the long-term memory module to the working memory module is "credit card loss". Based on this event, the working memory module can determine that the user need is "report the loss of the credit card to avoid or reduce financial losses". Of course, the identification between events and user needs can also be processed by the long-term memory module and output to the working memory module, which is not limited in the present application.

[0231] In the embodiment of the present application, the long-term memory module stores not only the prior emotion features corresponding to different events, but also the user needs corresponding to different events. For example, based on the event of the user's credit card being lost, the user's long-term memory module determines that the user's need is "report the loss of the credit card to avoid or reduce monetary loss" and outputs it to the working memory module.

[0232] Exemplarily, the primary decision module can obtain the purpose of the user's interaction with the digital human (i.e., the electronic device) based on the user's needs, that is, determine the interaction target. In addition, the primary decision module determines the emotional target (i.e., the emotion of the digital human) based on the user's emotions. The interaction target is used to provide the user with the required service, and the emotional target is used to meet the user's emotional needs.

[0233] Exemplarily, the primary decision module may pre-store an interaction target list (it may also be in other storage forms, which is not limited in this application), and the interaction target list includes different user needs (i.e., events) and corresponding interaction targets. The primary decision module may obtain the corresponding interaction target from the interaction target list based on the user needs input by the working memory module. Optionally, if the interaction target list does not include the corresponding user needs, the staff may be reminded to handle it.

[0234] For example, taking the above example, the user demand of user 1 is "report the loss of credit card to avoid or reduce financial loss". Accordingly, the primary decision-making module can query the interaction target list based on the user demand, and obtain the corresponding interaction target of "guiding the user to complete the credit card loss reporting process".

[0235] In the embodiments of this application, only the financial service scenario is used as an example for illustration. In other embodiments, different scenarios, user needs and emotions expressed by users are also different. The various goals and implementation methods determined by the primary decision module and the secondary decision module are different and can be set according to actual needs. This application does not limit them.

[0236] Exemplarily, the primary decision module may also be pre-set with an emotion target list (it may also be other storage forms, which is not limited in this application). The primary decision module may obtain the corresponding emotion target by searching the emotion target list based on the user emotions (i.e., the target emotion feature set) input by the working memory module.

[0237] For example, the primary decision module obtains the target emotional feature set {sadness, panic} input by the working memory module. The primary decision module can determine that the corresponding emotional goal is "relieving the user's sadness and panic" by searching the emotional target list.

[0238] For example, Figure 8 As shown, the primary decision module outputs the determined interaction goals and emotional goals to the secondary decision module as input of the secondary decision module.

[0239] S506, the electronic device determines a method for realizing the interaction target and a method for expressing the emotion target

[0240] Exemplarily, the secondary decision module in the electronic device can determine the implementation method (which can also be understood as a solution) of the interaction target based on the interaction target determined by the primary decision module, and can determine the expression method corresponding to the emotional target based on the emotional target.

[0241] Optionally, in an embodiment of the present application, the secondary decision module may further obtain user characteristics and the interaction environment, and determine the method of implementing the interaction goal in combination with the user characteristics and the interaction environment, so as to better fit the needs of the user in the current scenario and meet the diverse needs of different users in different scenarios.

[0242] Among them, user characteristics are used to describe the user's portrait, including but not limited to: age characteristics, gender characteristics, etc. As mentioned above, the user characteristics can be obtained by the perception module and output to the working memory module, and the working memory module can output the user characteristics to the secondary decision module.

[0243] Among them, the interactive environment is used to describe the characteristics of the environment in which the electronic device is located. Optionally, the interactive environment can be recognized by the perception module based on the image or audio recognized by the camera, and the perception module can be transmitted to the working memory module, and output by the working memory module to the secondary decision module. Optionally, the interactive environment can also be pre-set in the secondary decision module, or the secondary decision module can provide an interactive environment selection interface, which is selected by the user, and this application does not limit it. For example, in the financial hall scenario, the secondary decision module in the vertical screen can pre-set the interactive environment, that is, the financial service hall. For another example, in an online scenario, the secondary decision module in the mobile phone can provide an interactive environment selection interface for user selection. The user can choose a public environment or a private environment, etc., which is not limited by this application.

[0244] For example:

[0245] 1) The user feature description obtained by the secondary decision module is: a middle-aged or elderly woman. Accordingly, the secondary decision module can determine that the implementation method of the interactive target is the "senior version" based on the pre-settings (the correspondence between different user features and the implementation methods of the interactive targets can be set according to different scenarios, which is not limited in this application). Among them, the Chinese font of the interactive interface of the "senior version" is larger, and the process avoids complex information interfaces such as multiple clicks, so as to avoid the inconvenience caused to users by presbyopia and hand tremors.

[0246] 2) The secondary decision module obtains that the offline interaction environment is the financial service hall, and determines that the corresponding interaction target implementation method is "public environment", which is not suitable for presenting the user's privacy information on the screen.

[0247] Correspondingly, based on the above decision results, the secondary decision module obtains the implementation method of the interaction goal: calling the elderly-friendly version of the SDK for reissuing a credit card, wherein the elderly-friendly version of the SDK has features such as large font size, large click buttons, and text combined with voice broadcasting. In addition, this version hides sensitive information or privacy information when displayed.

[0248] Exemplarily, the secondary decision module can determine the expression corresponding to the emotional target based on the emotional target. Optionally, the secondary decision module can also pre-save the expression corresponding to different emotional targets. For example, if the user emotion (i.e., the target emotional feature set) is {sadness, panic}, then the corresponding emotional target is to alleviate the user's sadness and panic, and the corresponding expression can include: moderate comfort and clear willingness to help. The expressions in this application are only for illustration, and the correspondence between different expressions and emotional targets can be pre-set according to actual needs, and this application does not limit it.

[0249] Optionally, the secondary decision module can further determine the expression method corresponding to the emotional target based on the emotional target, the interactive environment and the user characteristics. For example, when the secondary decision module determines that the digital person's emotional expression is to appease based on the emotional target, and determines that the user is middle-aged or elderly based on the user characteristics and the interactive environment, and the noise in the interactive environment is relatively loud, it can be determined that when the digital person expresses emotions, the volume can be increased so that the user can hear clearly.

[0250] Exemplarily, the emotional expressions of digital humans include, but are not limited to: facial expressions, language content (i.e., text), voice, body language, etc. The secondary decision-making module can store specific content corresponding to different expressions. For example, the conversation content of the digital human is generated by a language model, and contains texts with two components: moderate comfort and clear willingness to help. For example, the text content of the moderate comfort component can be "Please don't worry yet / You don't have to worry / Please don't worry / You don't have to worry about this", etc. The text content of the clear willingness to help can be "I'll help you now / I'll ​​solve it for you right away / I'll ​​help you right away", etc.

[0251] In the embodiment of the present application, since each emotional component can be expressed in multiple ways, the digital human's response is not fixed and mechanical, but a diverse and flexible response that contains emotional target expression components.

[0252] Exemplarily, as described above, the secondary decision module can also determine the corresponding facial expressions and voice, body language, etc. For example, the secondary decision module can determine that when the digital person expresses the text "Please don't worry", the corresponding facial expressions, body language, and tone (i.e., voice rhythm) are all comforting expressions. For another example, when the secondary decision module determines that when the digital person expresses the text "I will help you with it now", the corresponding facial expressions, voice, body language, etc. are helpful expressions.

[0253] For example, the secondary decision module is based on the emotional goal (i.e., moderate comfort and clear willingness to help). By querying various expressions saved locally, the content can be determined to include: the text is "Please don't worry / You don't have to worry / Please don't worry / You don't have to worry about this", etc., and the corresponding facial expression is comforting, the body expression is comforting, and the voice rhythm expression is comforting voice rhythm expression. The text content that clearly indicates the willingness to help can be "I will help you now / I will solve it for you right away / I will help you right away", etc., and the corresponding facial expression is helping facial expression, the body expression is helping body expression, and the voice rhythm expression is helping voice rhythm expression.

[0254] Exemplarily, the secondary decision module may output the final decision to the execution module, so that the execution module specifically implements the interaction goal realization method and emotional expression.

[0255] S507, the electronic device executes a method for realizing the interaction target and a method for expressing the emotion target.

[0256] For example, Fig. 9 As shown, the secondary decision module outputs the implementation method of the interaction target and the expression method of the emotional target to the execution module. Accordingly, the execution module can call the corresponding task SDK based on the expression method of the interaction target. In addition, the execution module can generate the animation of the digital human based on the expression method of the emotional target. The process of generating animation includes but is not limited to: synthesizing facial expressions, synthesizing body movements, synthesizing speech, etc.

[0257] Specifically, the execution module can maintain task SDKs corresponding to different interactive target expressions. For example, the execution module receives the interactive target implementation method input by the secondary decision module, and the implementation method is used to instruct the call of the task SDK for reissuing a credit card. The task SDK for reissuing a credit card includes at least two versions, for example, a normal version and an aging-friendly version. As shown in Table 3:

[0258] Table 3

[0259]

[0260] Among them, S (font) Indicates the font size, S (button) Indicates the size of the button.

[0261] As shown in Table 3, the regular version of the task SDK for reissuing a credit card has a richer range of operability options and readable information cards, providing users with greater freedom and sufficient decision-making information. The aging-friendly version is designed for middle-aged and elderly groups, with large font size, large button area, less text information, and auxiliary voice broadcast, providing users with a simpler operation method.

[0262] Exemplarily, the execution module may further call the corresponding version based on the version information indicated by the expression mode of the interaction target. For example, as shown above, the expression mode of the interaction target determined by the secondary decision module is to call the SDK for the elderly version of the credit card reissue. Accordingly, the execution module calls the SDK for the elderly version of the credit card reissue based on the expression mode of the interaction target indicated by the secondary decision module.

[0263] In the embodiment of the present application, as described above, the secondary decision module can determine multiple emotional expressions, such as expressing comfort and expressing the willingness to help. Different emotional expressions correspond to different contents. For example, the secondary decision module can determine the text as "Please don't worry / You don't have to worry / Please don't worry / You don't have to worry about this", etc., and the corresponding facial expression is comforting, the facial expression is comforting, and the voice rhythm is comforting. The text content that clearly expresses the willingness to help can be "I'll help you now / I'll ​​solve it for you right here / I'll ​​help you right now", etc., and the corresponding facial expression is helping, the body expression is helping, and the voice rhythm is helping. Pronunciation expression. Correspondingly, the execution module can generate corresponding voice, body movements and expressions based on the decision of the secondary decision module, so that the emotional target components of the digital person include comfort and help, and the order is to first match the speech content related to comfort with the comfort expression, and then match the speech content related to help with the help expression. For example, when the digital person says "Please don't worry", his voice, expression and body all express comfort emotions. For another example, when the digital human says "I'll help you now," its voice, expression, and body language all express its willingness to help.

[0264] Exemplarily, when synthesizing the expression of the digital human, the execution module may utilize the currently disclosed Facial Action Coding System (FACS) technology to parameterize and generate the facial action unit (AU) of the digital human.

[0265] Specifically, the execution module sets a ratio parameter for each AU of the digital human face, which can be continuously adjusted from 0% to 100%. The AU parameter is determined by the emotional target component. The execution module can maintain AU parameters corresponding to different emotional components. As shown in Table 4:

[0266] Table 4

[0267] Emotional Components AU parameters comfort 100%AU1+20%AU4+20%AU12+10%AU25+10%AU26 help 50%AU1+50%AU2+50%AU6+80%AU12+10%AU25 Apologize 80%AU1+60%AU4+60%AU7+60%AU15+10%AU43 grateful 40%AU1+60%AU6+80%AU12+20%AU25+40%AU26

[0268] The AUs and their distributions involved in the embodiments of the present application are merely examples of suitability, and are not limited in the present application. As shown in Table 4, the execution module can adjust the amplitude of each AU in the face of the digital human to achieve different emotional expressions.

[0269] It should be noted that in the embodiment of the present application, the above parameters have a 10% variation range during the digital human's speech. For example, assuming that AU1 represents the inner eyebrow raising, 100% AU1 represents the inner eyebrow raising amplitude of 100%. During the digital human's speech, the inner eyebrow raising amplitude of the digital human fluctuates between 90% and 100%, making the digital human's expression more vivid.

[0270] The following is a detailed description of the contents in Table 4 using a specific example:

[0271] like Fig.10 As shown, the comfort emotion is expressed as: 100% AU1+20% AU4+20% AU12+10% AU25+10% AU26.

[0272] AU1-Inner Eyebrow Raise: The inner part of the eyebrow is raised upward to convey care and sympathy. The parameter is set to 100%, which means the inner eyebrow is raised by 100%.

[0273] AU4-Eyebrow Downward: Eyebrows are downward, which can convey seriousness and concern. The parameter is set to 20%, which means that the eyebrows are depressed by 20%.

[0274] AU12- Mouth Corner Stretch: The corners of the mouth move upward and outward, which can convey warmth and friendliness. The parameter is set to 20%, which means that the mouth corners are stretched by 20%.

[0275] AU25 - Lips Parted: Lips parted to convey openness and empathy. The parameter is set to 10%, which means the lips are parted by 10%.

[0276] AU26-Jaw Drop: The jaw drops downward, which can convey surprise or concern. The parameter is set to 10%, which means the jaw drops by 10%.

[0277] like Fig.11 As shown, the helpful emotion expression is: 50% AU1+50% AU2+50% AU6+80% AU12+10% AU25.

[0278] AU1-Inner Eyebrow Raise: The inner part of the eyebrow is raised upward to convey attention and concern for the user's needs. The parameter is set to 50%, that is, the inner eyebrow is raised by 50%.

[0279] AU2-Outer Eyebrow Raising: The outer eyebrows are raised upwards to convey friendliness and willingness to communicate with the user. The parameter is set to 50%, which means the outer eyebrow is raised by 50%.

[0280] AU6-Cheek Lift: Lifting the cheeks upward can convey warmth and friendliness. The parameter setting is 50%, which means the cheeks are lifted by 50%.

[0281] AU12- Mouth corner stretch: The corners of the mouth move upward and outward to convey a willingness to help. The parameter setting is 80%, which means the mouth corners are stretched by 80%.

[0282] AU25-Lips Parted: Lips parted can convey openness and willingness to listen to others. The parameter setting is 10%, which means the lips are parted by 10%.

[0283] like Fig.12 As shown, the apology emotion is expressed as: 100% AU1+10% AU4+60% AU7+60% AU15+10% AU43.

[0284] AU1-Inner Eyebrow Raise: The inner part of the eyebrow is raised upwards, which can convey sadness when used with AU4. The parameter is set to 100%, which means that the inner eyebrow is raised by 100%.

[0285] AU4-Eyebrows down: Eyebrows down, can convey the seriousness and sincerity of the apology. The parameter is set to 10%, that is, the eyebrow down is 10%.

[0286] AU7-Eyelid Tightening: Eyelid tightening can convey shame and regret. The parameter setting is 60%, which means the eyelid tightening is 60%.

[0287] AU15- Mouth corners pressed down: The downward movement of the mouth corners can convey sadness. The parameter setting is 60%, that is, the mouth corners are pressed down by 60%.

[0288] AU43-Eyes Closed: Eyes closed downwards can convey shame and regret. The parameter is set to 10%, which means the eyes close by 10%.

[0289] like Fig.13 As shown, the emotion of gratitude is expressed as: 40% AU1+60% AU6+80% AU12+20% AU25+40% AU26.

[0290] AU1-Inner Eyebrow Raise: The inner part of the eyebrow is raised upward to convey excitement. The parameter is set to 40%, which means the inner eyebrow is raised by 40%.

[0291] AU6-Cheek Lift: Lifting the cheeks upward can convey a happy and positive mood. The parameter setting is 60%, which means the cheeks are lifted by 60%.

[0292] AU12- Mouth Corner Stretch: The mouth corners move upward and outward, which can convey happiness and gratitude. The parameter setting is 80%, that is, the mouth corners are stretched by 80%.

[0293] AU25-Lips Parted: Lips parted to convey openness. The parameter is set to 20%, which means the lips are parted by 20%.

[0294] AU26-Jaw Drop: The jaw drops downward to convey surprise or delight. The parameter setting is 40%, which means the jaw drops by 40%.

[0295] Exemplarily, the specific method of synthesizing speech by the execution module is: the execution module generates corresponding speech based on the text input by the secondary decision module, such as "Please don't worry first". Among them, the prosodic features of the speech are set according to the emotional expression method indicated by the secondary decision module (which can also be understood as emotional components, such as comfort, help, etc.). Table 5 shows the correspondence between speech prosody and emotional components.

[0296] Table 5

[0297]

[0298]

[0299] Among them, speech prosody includes three characteristic parameters: audio, duration and energy. For different emotional components, different emotions can be expressed by adjusting the above three characteristic parameters. Among them, the parameter in a calm state is 1, and the values ​​in the table are the ratios of the parameters in a calm state.

[0300] Exemplarily, the specific method for the execution module to synthesize the body movements of the digital human is as follows: the execution module can set the activity speed and amplitude of the digital human's body (including the head, torso, hands, etc.) based on different emotional components. Table 6 shows the correspondence between body movements and emotional components.

[0301] Table 6

[0302]

[0303] Among them, the three gears of slow, medium and fast in the speed of the motion feature can correspond to the preset speed range respectively, which can be set according to actual needs, and this application does not limit it. And the large and small gears of the amplitude of the motion feature can also set the corresponding preset amplitude range, which is not limited in this application. The action type, speed and amplitude in the embodiments of this application are only for schematic illustration, and can be set according to different scene requirements, which is not limited in this application.

[0304] It should be noted that, as mentioned above, the text, body movements and expressions of the digital human are coordinated with each other. For example, when the digital human needs to express comfort, the emotional components corresponding to the text, body movements, expressions and voice rhythm output by the digital human are all "comfort". Based on the emotional components that need to be expressed, when generating the animation of the digital human, the execution module can synthesize the expressions, body movements and voice rhythm of the digital human in the animation based on the corresponding relationship between each emotional component in the above table and the corresponding expression method.

[0305] For example, let's take the above example of user 1 applying for a credit card in a financial service scenario. For example, user 1 is an elderly user. Figure 3In the scenario shown, the user stands on the screen and says to the digital person "I lost my credit card". The digital person (specifically, an electronic device) can collect the user's voice and picture. The electronic device obtains the user's words "I lost my credit card", which is the active interaction information. Based on the active interaction information, the electronic device recognizes that the corresponding text is "I lost my credit card". The electronic device can obtain the user's needs and prior emotional features based on the text information corresponding to the active interaction information. Among them, the user's demand is "report the loss of credit card to avoid or reduce monetary losses", and the prior emotional features are {sadness, panic}. At the same time, the electronic device can determine the posterior emotional features, such as {sadness}, based on the collected non-active interaction information such as user pictures and voice rhythm. The electronic device can determine the user's emotion (i.e., the target emotional feature) as {sadness} based on the prior emotional features and the posterior emotional features. Correspondingly, the electronic device further determines the implementation method (i.e., the solution) of the interaction target based on user needs, interaction environment, user characteristics, etc.: calling the SDK of the aging-friendly version of the credit card reissue. Among them, the SDK for the elderly version of the credit card replacement has the following features: large font size, large click button, text with voice broadcast, and no more than 3 options. In addition, based on the user's emotions, the electronic device determines the emotional goals (which can also be understood as emotional components) of the digital person as comfort and help. Based on the emotional goals, the electronic device synthesizes the digital person's expressions, body movements, voice rhythms, output text, etc. to generate animations.

[0306] Electronic devices display animations. Specifically, as mentioned above, the expression of the emotional goals of digital humans requires two components, including: moderate comfort and clear willingness to help. Therefore, the rhythmic features of the digital human's facial expressions, body movements, and voice all contain these two components and are synchronized in presentation time. Specifically, when the digital human expresses the comfort component, the generated language text may be "Please don't worry, you don't have to worry, please don't worry, you don't have to worry about this", and the facial expression parameters are "40% AU1+60% AU4+50% AU12+10% AU25+10% AU26", using slow, small-amplitude, head-forward body movements, and the corresponding voice rhythmic feature parameters are "fundamental frequency: 1; duration: 1.2; energy: 1-4". For example, when a digital human expresses a helping component, the generated language text may be "I'll help you now, I'll solve it for you right away, I'll help you right away", and the corresponding facial expression parameters are "80% AU1+80% AU2+50% AU6+80% AU12+10% AU25", the torso is slightly leaning forward, the right hand is pointing to itself and the user in turn, and the body movements are medium-speed and large-scale. The corresponding speech rhythm feature parameters are "fundamental frequency: 1; duration: 1; energy: 1-4".

[0307] For example, after the digital person expresses his willingness to help, the SDK for the elderly can be called to reissue a credit card. Fig.14 As shown, in the display interface of the vertical screen, the digital human's mouth opens and closes in the animation of the digital human to indicate that the digital human is speaking. At the same time, the electronic device outputs the digital human's voice to inform the user that "you can click the [Credit Card Loss Report] button to report the loss". In addition, in the aging-friendly version, when the clickable button control appears for the first time, an animation with a finger click is used to prompt the user to click to operate. After guiding the user to click, the digital human tells the user "Please check your credit card information", and the clickable button control uses the same control appearance as the first time, so that the user knows that it can be clicked here. At the same time, the options are combined with symbols familiar to the user (such as "×" for error; "√" for correct) to assist the user in making judgments. Among them, when guiding the user through the credit card loss report process in class, the digital human's voice rhythm, expression, body movements, etc. can continuously express the willingness to help.

[0308] It is understandable that, in order to realize the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0309] In one example, Fig.15 A schematic block diagram of a device 1500 according to an embodiment of the present application is shown. The device 1500 may include: a processor 1501 and a transceiver / transceiver pin 1502 , and optionally, a memory 1503 .

[0310] The components of the device 1500 are coupled together via a bus 1504, wherein the bus 1504 includes a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, all buses are referred to as bus 1504 in the figure.

[0311] Optionally, the memory 1503 may be used for the instructions in the aforementioned method embodiment. The processor 1501 may be used to execute the instructions in the memory 1503, and control the receiving pin to receive a signal, and control the sending pin to send a signal.

[0312] The apparatus 1500 may be the electronic device or a chip of the electronic device in the above method embodiment.

[0313] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0314] This embodiment further provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.

[0315] This embodiment also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the method in the above-mentioned embodiment.

[0316] In addition, an embodiment of the present application also provides a device, which may specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor may execute the computer-executable instructions stored in the memory so that the chip executes the methods in the above-mentioned method embodiments.

[0317] Among them, the electronic device, computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above and will not be repeated here.

[0318] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A human-computer interaction method, characterized in that: include: Acquire active interaction information input by a user, where the active interaction information is used to represent the needs of the user in a target scenario; Based on the active interaction information, querying an experience knowledge base for a first emotion feature set corresponding to the active interaction information; In the process of the user inputting the active interaction information, collecting the user's inactive interaction information, wherein the inactive interaction information is used to characterize the user portrait of the user; Determining a second set of emotion features based on the non-active interaction information; Determining the emotion of the user in the target scene based on the first emotion feature set and the second emotion feature set; Based on the emotion of the user in the target scene, an emotion expression feature of the digital human is determined, where the emotion expression feature includes at least one of facial expression, body movement, and voice.

2. The method according to claim 1, characterized in that After obtaining the active interaction information input by the user, the method further includes: Determining a user demand of the user based on the active interaction information; Based on the user needs, determine the target solution; Correspondingly, after determining the emotional expression features of the digital human based on the first emotion feature set and the second emotion feature set, the method further includes: Based on the target solution and the emotional expression characteristics, generating a target animation of the digital human; Play the target animation.

3. The method according to claim 2, characterized in that The step of generating a target animation of the digital human based on the target solution and the emotional expression feature comprises: Acquire scene environment features in the target scene, and / or, acquire user features of the user based on the inactive interaction information, wherein the user features belong to the user portrait; Determining a display mode of the animation part corresponding to the target solution based on the scene environment characteristics and / or the user characteristics; A target animation of the digital human is generated based on the target solution, the display mode corresponding to the target solution, and the emotional expression feature.

4. The method according to claim 2, characterized in that: The step of generating a target animation of the digital human based on the target solution and the emotional expression feature comprises: Based on the emotion expression features, the facial expression of the digital human is synthesized.

5. The method according to claim 4, characterized in that The step of synthesizing the facial expression of the digital human based on the emotion expression feature comprises: Based on the emotion expression feature, determining an adjustment parameter of at least one facial unit of the digital human; The at least one facial unit is adjusted based on the adjustment parameter of the at least one facial unit.

6. The method according to claim 1, characterized in that Determining the emotion of the user in the target scene based on the first emotion feature set and the second emotion feature set includes: Based on the first emotion feature set and the second emotion feature set, a target emotion feature set is determined; wherein the target emotion feature set includes at least one emotion feature, and the at least one emotion feature is used to represent the emotion of the user when he is in the target scene and inputs the active interaction information.

7. The method according to claim 6, characterized in that The target emotion feature set is the intersection of the first emotion feature set and the second emotion feature set; or, If the intersection of the first emotion feature set and the second emotion feature set is zero, and the first emotion feature set is not empty, then the target emotion feature set is the first emotion feature set; or, If the intersection of the first emotion feature set and the second emotion feature set is zero, and the first emotion feature set is empty, then the target emotion feature set is the second emotion feature set.

8. The method according to claim 7, characterized in that If the intersection of the first emotion feature set and the second emotion feature set is zero, and the first emotion feature set is not empty; Alternatively, if the intersection of the first emotion feature set and the second emotion feature set is zero, and the first emotion feature set is empty, the method further includes: If the second emotion feature set is not empty, the experience knowledge base is updated based on the second emotion feature set.

9. The method according to claim 1, characterized in that: The step of searching, based on the active interaction information, an experience knowledge base for a first set of emotion features corresponding to the active interaction information includes: Based on the active interaction information, a target event is acquired, where the target event matches the needs of the user; Based on the target event, the first emotion feature set is queried in the experience knowledge base, where the experience knowledge base includes a correspondence between events and emotion feature sets.

10. The method according to claim 1, characterized in that The active interaction information includes at least one of the following: voice information, text information, and gesture information.

11. The method according to claim 1, characterized in that: The non-active interaction information includes at least one of the following: user's appearance feature information and speech rhythm feature information.

12. An electronic device, characterized in that: include: one or more processors; Memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the electronic device performs the following steps: Acquire active interaction information input by a user, where the active interaction information is used to represent the needs of the user in a target scenario; Based on the active interaction information, querying an experience knowledge base for a first emotion feature set corresponding to the active interaction information; In the process of the user inputting the active interaction information, collecting the user's inactive interaction information, wherein the inactive interaction information is used to characterize the user portrait of the user; Determining a second set of emotion features based on the non-active interaction information; Determining the emotion of the user in the target scene based on the first emotion feature set and the second emotion feature set; Based on the emotion of the user in the target scene, an emotion expression feature of the digital human is determined, where the emotion expression feature includes at least one of facial expression, body movement, and voice.

13. The electronic device according to claim 12, characterized in that: When the computer program is executed by the one or more processors, the electronic device performs the following steps: Determining a user demand of the user based on the active interaction information; Based on the user needs, determine the target solution; Based on the target solution and the emotional expression characteristics, generating a target animation of the digital human; Play the target animation.

14. The electronic device according to claim 13, characterized in that: When the computer program is executed by the one or more processors, the electronic device performs the following steps: Acquire scene environment features in the target scene, and / or, acquire user features of the user based on the inactive interaction information, wherein the user features belong to the user portrait; Determining a display mode of the animation part corresponding to the target solution based on the scene environment characteristics and / or the user characteristics; A target animation of the digital human is generated based on the target solution, the display mode corresponding to the target solution, and the emotional expression feature.

15. The electronic device according to claim 13, characterized in that: When the computer program is executed by the one or more processors, the electronic device performs the following steps: Based on the emotion expression features, the facial expression of the digital human is synthesized.

16. The electronic device according to claim 15, characterized in that: When the computer program is executed by the one or more processors, the electronic device performs the following steps: Based on the emotion expression feature, determining an adjustment parameter of at least one facial unit of the digital human; The at least one facial unit is adjusted based on the adjustment parameter of the at least one facial unit.

17. The electronic device according to claim 12, characterized in that: When the computer program is executed by the one or more processors, the electronic device performs the following steps: Based on the first emotion feature set and the second emotion feature set, a target emotion feature set is determined; wherein the target emotion feature set includes at least one emotion feature, and the at least one emotion feature is used to represent the emotion of the user when he is in the target scene and inputs the active interaction information.

18. The electronic device according to claim 17, characterized in that: The target emotion feature set is the intersection of the first emotion feature set and the second emotion feature set; or, If the intersection of the first emotion feature set and the second emotion feature set is zero, and the first emotion feature set is not empty, then the target emotion feature set is the first emotion feature set; or, If the intersection of the first emotion feature set and the second emotion feature set is zero, and the first emotion feature set is empty, then the target emotion feature set is the second emotion feature set.

19. The electronic device according to claim 18, characterized in that: If the intersection of the first emotion feature set and the second emotion feature set is zero, and the first emotion feature set is not empty; Alternatively, if the intersection of the first emotion feature set and the second emotion feature set is zero, and the first emotion feature set is empty, the method further includes: If the second emotion feature set is not empty, the experience knowledge base is updated based on the second emotion feature set.

20. The electronic device according to claim 12, characterized in that: When the computer program is executed by the one or more processors, the electronic device performs the following steps: Based on the active interaction information, a target event is acquired, where the target event matches the needs of the user; Based on the target event, the first emotion feature set is queried in the experience knowledge base, where the experience knowledge base includes a correspondence between events and emotion feature sets.

21. A computer storage medium, characterized in that The method comprises computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 11.

22. A chip, characterized in that: It comprises one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send the signal to the processor, the signal comprising a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the method described in any one of claims 1 to 11.

Images