Emotion expression method and device, pet robot, and storage medium

Abstract

This invention provides a method, device, pet robot, and storage medium for expressing emotions, relating to the field of data processing technology. The method includes: acquiring user expression data; performing emotion recognition on dialogue content data to obtain a dialogue emotion recognition result, and recognizing auxiliary expression data to obtain an auxiliary recognition result; determining the dialogue emotion recognition result as the user's emotion result when the dialogue emotion recognition result meets preset acceptance conditions and the emotion pointed to by the auxiliary recognition result is consistent with the emotion pointed to by the dialogue emotion recognition result; determining the user's emotion result based on the auxiliary recognition result when the dialogue emotion recognition result does not meet preset acceptance conditions; determining the anthropomorphic emotion of the interactive device based on the user's emotion result, determining device expression data based on the anthropomorphic emotion, and outputting the device expression data; wherein the device expression data is used to characterize the anthropomorphic emotion of the interactive device in response to the user's expression data.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, device, pet robot, and storage medium for expressing emotions. Background Technology

[0002] Child companion robots are intelligent interactive devices designed for young children. Through functions such as voice dialogue, facial expression interaction, motion sensing, and content playback, they provide children with emotional companionship, early education, and safety monitoring. To enhance interactivity, some pet robots in related technologies have added movable mechanical limbs, such as swinging ears and tails. Some pet robots with walking capabilities can even raise their forelimbs to greet users.

[0003] However, pet robots in related technologies lack the ability to perceive and express emotions. On the one hand, the basis for the device's judgment of the user's emotions is relatively singular, and the judgment result is easily interfered with by factors such as the user turning their head, facial occlusion, changes in lighting, brief facial expressions, and the literal meaning of words, leading to deviations in the device's judgment of the user's true emotions. On the other hand, even when the judgment is biased, the device still responds directly, making the response content mismatched with the user's true emotions, making it difficult to establish a deep emotional connection between the device and the user, and affecting the companionship experience. Summary of the Invention

[0004] This invention provides a method, device, pet robot, and storage medium for expressing emotions, in order to solve the problems of insufficient accuracy in judging user emotions and mismatch between response content and user's true emotions in related technologies.

[0005] This invention provides a method for expressing emotions, applied to interactive devices, the method comprising: Acquire user expression data; wherein, the user expression data includes dialogue content data and auxiliary expression data, and the auxiliary expression data is data used to characterize user emotions in the user expression data other than the dialogue content data; Emotion recognition is performed on the dialogue content data to obtain the dialogue emotion recognition result, and the auxiliary expression data is recognized to obtain the auxiliary recognition result; If the dialogue emotion recognition result meets the preset acceptance conditions, and the emotion pointed to by the auxiliary recognition result is consistent with the emotion pointed to by the dialogue emotion recognition result, the dialogue emotion recognition result is determined as the user's emotion result; wherein, the preset acceptance conditions include: the dialogue content data contains the user's expression of their own emotions, and the expression is not a negative expression, a question expression, or a paraphrase expression. If the dialogue emotion recognition result meets the preset acceptance conditions, and the emotion pointed to by the auxiliary recognition result is inconsistent with the emotion pointed to by the dialogue emotion recognition result, a confirmation question is output to the user, and based on the user's response to the confirmation question, the dialogue emotion recognition result is maintained as the user's emotion result, or the user's emotion result is redefined; before the response is obtained, the dialogue emotion recognition result is used as the user's emotion result. If the dialogue emotion recognition result does not meet the preset acceptance conditions, the user's emotion result is determined based on the auxiliary recognition result; Based on the user's emotion result, the anthropomorphic emotion of the interactive device is determined, and based on the anthropomorphic emotion, device expression data is determined and output; wherein, the device expression data is used to characterize the anthropomorphic emotion of the interactive device in response to the user's expression data.

[0006] According to a method for expressing emotions provided by the present invention, the method further includes: If the emotion indicated by the auxiliary recognition result is consistent with the emotion indicated by the dialogue emotion recognition result, the feedback intensity corresponding to the device expression data is determined based on the obviousness of the emotion expression in the auxiliary recognition result. The more pronounced the emotional expression, the stronger the feedback.

[0007] According to a method for expressing emotions provided by the present invention, when the dialogue emotion recognition result does not meet the preset acceptance conditions, determining the user's emotion result based on the auxiliary recognition result includes: If the dialogue content corresponding to the dialogue content data is a short sentence with incomplete semantics, the user emotion result is supplemented and determined by combining the auxiliary recognition result, or the previous valid user emotion result is retained. If the dialogue content data is not obtained, a temporary judgment result is determined based on the auxiliary recognition result, and feedback with an intensity lower than a preset intensity is output based on the temporary judgment result; after new dialogue content data is obtained, the user emotion result is updated based on the dialogue emotion recognition result corresponding to the new dialogue content data.

[0008] According to a method for expressing emotions provided by the present invention, the auxiliary expression data includes facial image data, which includes multiple frames of facial images acquired consecutively within a preset time period; the auxiliary expression data is then identified to obtain an auxiliary identification result, including: Emotion recognition is performed on each frame of the multi-frame face images to obtain multiple single recognition results; If the single recognition results obtained consecutively reach a preset number all belong to the same preset facial emotion category, the same preset facial emotion category will be used as the preliminary facial emotion recognition result. If any of the following face images in the multi-frame face images is too small, blurry, occluded, or has its head turned beyond a preset angle, then the face image containing at least one of these conditions is an invalid image, and the invalid image is not used to update the preliminary face emotion recognition result.

[0009] According to a method for expressing emotions provided by the present invention, the method further includes: If the preliminary facial emotion recognition result meets the secondary confirmation conditions, the preliminary facial emotion recognition result is temporarily suspended, and a question to confirm the emotion is output to the user, or facial emotion recognition continues within a preset observation period; wherein, the secondary confirmation conditions include at least one of the following: different preset facial emotion categories change repeatedly within the preset period, preset easily confused emotion categories appear alternately, the preliminary facial emotion recognition result is inconsistent with the dialogue content data, or the collected facial image is an invalid image; If the user's response indicates their own emotion, or if the single recognition results obtained in the continued recognition process reach a preset number of times and belong to the same preset facial emotion category and do not conflict with the dialogue content data, the confirmed preset facial emotion category will be used as the auxiliary recognition result. If the user does not respond to their own emotion within the preset confirmation time and the recognition continues without obtaining a single recognition result belonging to the same preset facial emotion category, the previous valid user emotion result will be retained, or a neutral response will be output.

[0010] According to a method for expressing emotions provided by the present invention, the dialogue content data includes user voice data or user-input text data; performing emotion recognition on the dialogue content data to obtain a dialogue emotion recognition result includes: The user's voice data is converted into text to obtain the current dialogue text, or the text data input by the user is used as the current dialogue text; Emotion recognition is performed by combining the current dialogue text with the content of at least one previous round of historical dialogue to obtain the dialogue emotion recognition result.

[0011] According to a method for expressing emotions provided by the present invention, emotion recognition is performed by combining the current dialogue text and the content of at least one previous round of historical dialogue to obtain the dialogue emotion recognition result, including: When multiple emotions appear simultaneously in the current dialogue text, and the current dialogue text contains the user's expression of their own emotions, the emotion corresponding to the user's expression of their own emotions is taken as the dialogue emotion recognition result. If multiple emotions appear simultaneously in the current dialogue text, and there is no expression of the user's own emotions in the current dialogue text, the emotion corresponding to the main purpose expressed at the end of the current dialogue text shall be taken as the dialogue emotion recognition result. If the emotion cannot be determined based on the current dialogue text, the dialogue emotion recognition result is determined to be in a pending confirmation state; wherein, the dialogue emotion recognition result in the pending confirmation state does not meet the preset acceptance conditions.

[0012] According to the present invention, the negative expression is a statement that contains a negative component in the emotion word, the question expression is a statement that asks the interactive device or a third party about the emotion, and the paraphrased expression is a paraphrasing of the emotion in the story content, lyrics, or third party's speech.

[0013] According to a method for expressing emotions provided by the present invention, the auxiliary expression data includes action command data, which is action data of a user acting on an interactive device detected by a sensor; the auxiliary expression data is then identified to obtain an auxiliary identification result, including: Based on the severity, number of times, and duration of the action command data, the action command data is divided into ordinary interaction signals or strong action signals, which serve as the auxiliary recognition result. Neither the ordinary interactive signal nor the stronger action signal independently determines the user's emotional outcome.

[0014] According to an emotion expression method provided by the present invention, the dialogue emotion recognition result is one of a plurality of preset dialogue emotion categories, the plurality of preset dialogue emotion categories including: happy, angry, afraid, sad, coquettish, confused, curious and gentle; The preset facial emotion categories include happy, sad, angry, surprised, afraid, disgusted, and normal.

[0015] According to a method for expressing emotions provided by the present invention, based on the anthropomorphic emotions, device expression data is determined, including: Based on the pre-established mapping relationship between anthropomorphic emotions and expression methods, the expression method corresponding to the anthropomorphic emotion is determined as the device expression data; In the mapping relationship, the anthropomorphic emotions are divided into multiple intensity levels according to their intensity, and different intensity levels correspond to different expression methods. The expression methods include at least one of facial expression, body movement expression, voice tone expression, and light expression. The higher the intensity of the anthropomorphic emotion, the more obvious the body movement expression.

[0016] According to a method for expressing emotions provided by the present invention, the device expressing data includes at least one of the following: When the user's emotional state is anger, the device's expression data includes lowering the volume, stopping the currently playing content, and expressing an apology; When the user's emotional outcome is fear or sadness, the device's expressive data includes comforting content output in a reassuring tone; In the case where the user's emotional outcome is confusion, the device expression data includes prior output content that has been reinterpreted in a simplified manner; When the user's emotional outcome is curiosity, the device's expressive data includes answers to the user's questions and extended content.

[0017] The present invention also provides an emotion expression device, the device comprising: An acquisition module is used to acquire user expression data; wherein, the user expression data includes dialogue content data and auxiliary expression data, and the auxiliary expression data is data used to characterize user emotions in the user expression data other than the dialogue content data; The recognition module is used to perform emotion recognition on the dialogue content data to obtain the dialogue emotion recognition result, and to recognize the auxiliary expression data to obtain the auxiliary recognition result; The determination module is configured to: determine the dialogue emotion recognition result as the user's emotion result when the dialogue emotion recognition result meets preset acceptance conditions and the emotion pointed to by the auxiliary recognition result is consistent with the emotion pointed to by the dialogue emotion recognition result; output a confirmation query to the user when the dialogue emotion recognition result meets the preset acceptance conditions and the emotion pointed to by the auxiliary recognition result is inconsistent with the emotion pointed to by the dialogue emotion recognition result, and maintain the dialogue emotion recognition result as the user's emotion result or redetermine the user's emotion result based on the user's response to the confirmation query; and determine the user's emotion result based on the auxiliary recognition result when the dialogue emotion recognition result does not meet the preset acceptance conditions. The preset acceptance conditions include: the dialogue content data contains a user's expression of their own emotions, and the expression is not a negative expression, a question expression, or a paraphrase expression. An expression module is used to determine the anthropomorphic emotion of the interactive device based on the user's emotion result, determine device expression data based on the anthropomorphic emotion, and output the device expression data; wherein, the device expression data is used to characterize the anthropomorphic emotion of the interactive device in response to the user's expression data.

[0018] The present invention also provides a pet robot, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the emotion expression method as described above.

[0019] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the emotion expression method as described above.

[0020] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the emotion expression method as described above.

[0021] This invention provides an emotion expression method, device, pet robot, and storage medium. By setting preset acceptance conditions defined by negative statements, interrogative statements, and paraphrased statements, dialogue content whose literal meaning is inconsistent with the user's true emotion is excluded from direct acceptance, avoiding misjudgments caused by directly judging based on emotion keywords. When the dialogue emotion recognition result is acceptable but conflicts with the auxiliary recognition result, a confirmation question is output first, and the dialogue emotion recognition result is maintained until a reply is received, avoiding erroneous responses caused by momentary expressions or single actions overshadowing the clear meaning of the dialogue. When the dialogue emotion recognition result is unacceptable, the auxiliary recognition result is used to supplement the judgment, forming a complete user emotion determination mechanism. Based on this, the anthropomorphic emotion of the interactive device is derived from the user's emotion result, and the corresponding device expression data is output, so that the response of the interactive device matches the user's true emotion, which is conducive to establishing a deep emotional connection between the device and the user. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating the emotion expression method provided by the present invention; Figure 2 This is a flowchart illustrating the process of determining user emotion results provided by the present invention; Figure 3 This is a flowchart illustrating the facial emotion recognition process provided by the present invention; Figure 4 This is a flowchart illustrating the dialogue emotion recognition process provided by the present invention; Figure 5 This is a schematic diagram illustrating the mapping relationship between anthropomorphic emotions and expression methods provided by the present invention; Figure 6 A schematic diagram of the structure of the emotion expression device provided by the present invention; Figure 7 This is a structural schematic diagram of the pet robot provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] The execution entity of the emotion expression method provided by the present invention can be a processor or controller built into the interactive device, a cloud server that communicates with the interactive device, or a control system composed of the interactive device and the cloud server working together.

[0026] For ease of explanation, the following embodiments use the processor built into the interactive device as the execution subject, and use a pet robot in a child companionship scenario as an example of the interactive device for explanation. The pet robot has mechanical limbs such as a microphone, camera, tactile sensor, speaker, facial expression display component, and movable ears.

[0027] Figure 1 This is a flowchart illustrating the emotion expression method provided by the present invention, as shown below. Figure 1 As shown, the method includes the following: Step 110: Obtain user expression data.

[0028] In this application, user expression data refers to data expressed by the user to the interactive device and acquired by the interactive device during the interaction between the user and the interactive device, including dialogue content data and auxiliary expression data.

[0029] Dialogue content data refers to the data corresponding to the content expressed by the user to the interactive device by speaking or typing, such as a sentence spoken by a child user to a pet robot; auxiliary expression data is data in the user's expression data other than dialogue content data, which is used to characterize the user's emotions. It can be one or more of the following: user facial state data, body movement data, operation data acting on the interactive device itself, and acoustic feature data of the speaking voice.

[0030] The acquisition method can be real-time acquisition through acquisition components such as microphones, cameras, and tactile sensors, or reading from the cache of the interactive device, or receiving from other devices through a communication interface. This embodiment does not limit the acquisition method.

[0031] Step 120: Perform emotion recognition on the dialogue content data to obtain the dialogue emotion recognition result, and perform recognition on the auxiliary expression data to obtain the auxiliary recognition result.

[0032] In this application, the dialogue emotion recognition result is used to characterize the emotions expressed by the user through the spoken content; the emotion recognition of the dialogue content data can be performed by a matching method based on an emotion lexicon and semantic rules, or by a trained text emotion classification model, which takes the text corresponding to the dialogue content data as input and the emotion category as output.

[0033] The auxiliary recognition results are used to characterize the emotion or emotion-related state pointed to by the auxiliary expression data. The recognition method is adapted to the type of auxiliary expression data. For example, facial state data can be processed by expression recognition, while body movement data and operation data can be processed by preset action feature analysis.

[0034] Step 130: Determine the user's emotion result based on the dialogue emotion recognition result and the auxiliary recognition result. Figure 2 This is a flowchart illustrating the user emotion result determination process provided by the present invention, as shown below. Figure 2 As shown, the determination process is divided into three scenarios based on whether the dialogue emotion recognition results meet the preset acceptance conditions.

[0035] The pre-defined acceptance criteria include the presence of user expressions of their own emotions in the dialogue content data, and these expressions are not negative, interrogative, or paraphrased. For example, when a user says "I'm so sad," it is a direct expression of their own emotions and meets the pre-defined acceptance criteria. However, the emotional words in "I'm not angry" are in a negative context, "Are you angry?" is a question directed at the other person, and "The little rabbit in the story is scared" is a paraphrase of the story content. None of these three meet the pre-defined acceptance criteria.

[0036] In the first scenario of this application, the dialogue emotion recognition result meets the preset acceptance conditions, and the emotion pointed to by the auxiliary recognition result is consistent with the emotion pointed to by the dialogue emotion recognition result. In this case, the dialogue emotion recognition result is determined as the user's emotion result.

[0037] Consistency means that both emotions point to the same emotional category, or that the emotions they point to are semantically compatible and do not conflict; for example, if a user says "I am so sad" and at the same time their facial expression shows sadness, the two are consistent, and the user's emotional result is determined to be sadness.

[0038] In the second scenario of this application, if the dialogue emotion recognition result meets the preset acceptance conditions and the emotion pointed to by the auxiliary recognition result is inconsistent with the emotion pointed to by the dialogue emotion recognition result, then a confirmation question is output to the user, and based on the user's response to the confirmation question, the dialogue emotion recognition result is maintained as the user's emotion result, or the user's emotion result is redefined.

[0039] Confirmation queries are statements used to verify a user's emotional state and are easy for the user to answer directly. For example, if a user smiles but says, "I'm really angry," the processor controls the pet robot to ask, "Are you angry right now?" If the user answers "confirm," the anger is maintained as the user's emotional state. If the user answers "deny," the user's emotional state is reassessed based on the content of the answer. Before receiving a response, the emotional state recognition result of the dialogue is used as the user's emotional state result, so that a single auxiliary recognition result will not immediately change the judgment made based on the explicit dialogue.

[0040] In the third scenario of this application, the dialogue emotion recognition result does not meet the preset acceptance conditions. In this case, the user's emotion result is determined based on the auxiliary recognition result. That is, when the dialogue content is insufficient to explain the user's own emotions, the emotion information reflected by the auxiliary expression data is used to supplement the judgment.

[0041] Step 140: Based on the user's emotion results, determine the anthropomorphic emotion of the interactive device, determine the device expression data based on the anthropomorphic emotion, and output the device expression data.

[0042] In this application, anthropomorphic emotion refers to the human-like emotional state presented by the interactive device in response to the user's emotions, so as to make the user perceive that the interactive device also has emotions; the determination method can be based on a pre-set corresponding rule, such as determining the anthropomorphic emotion as gentle when the user's emotional result is sadness, and determining the anthropomorphic emotion as apologetic when the user's emotional result is anger, or the anthropomorphic emotion can be output by the dialogue generation model in combination with the interaction context.

[0043] In this application, device expression data is output data used to characterize the anthropomorphic emotions of the interactive device in response to user expression data, and may include one or more of voice response data, facial expression display data, body movement control data, and lighting control data; the output methods include playing voice response data through a speaker, displaying facial expressions through an facial expression display component, driving mechanical limbs to perform actions through a driving component, and presenting corresponding lighting effects through a lighting component.

[0044] The method provided in this embodiment defines the credibility of dialogue emotion recognition results by setting preset acceptance conditions, and forms three determination paths according to whether to accept or not and whether it is consistent with the auxiliary recognition results. This ensures that literal emotion words that do not match the real emotion are not directly accepted, and that instantaneous auxiliary information does not cover the clear meaning of the dialogue, thereby improving the accuracy of the user's emotion judgment and making the anthropomorphic emotion expression output by the interactive device match the user's real emotion.

[0045] Optionally, the method further includes: If the emotion indicated by the auxiliary recognition result is consistent with the emotion indicated by the dialogue emotion recognition result, the feedback intensity corresponding to the device expression data is determined based on the obviousness of the emotion expression in the auxiliary recognition result. The more pronounced the emotional expression, the stronger the feedback.

[0046] In this application, when the emotion indicated by the auxiliary recognition result is consistent with the emotion indicated by the dialogue emotion recognition result, the processor determines the feedback intensity corresponding to the device expression data based on the obviousness of the emotion expression in the auxiliary recognition result.

[0047] The intensity of emotional expression can be measured by the strength of facial features, the amplitude and frequency of body movements, or the confidence score of recognition; the intensity of feedback can be reflected by the volume and emotional intensity of the voice response, the amplitude of mechanical body movements, and the duration of the reassurance content. The higher the intensity of emotional expression, the higher the intensity of feedback.

[0048] For example, when a child user says "I'm so sad" and their face shows obvious signs of sadness, the pet robot uses a more obvious comforting response; when the facial signs of sadness are not obvious, it uses a milder comforting response.

[0049] This embodiment uses auxiliary recognition results consistent with the dialogue results to adjust the feedback intensity, so that the auxiliary information can play a role in confirmation and rating without changing the emotion judgment conclusion, and make the response intensity of the interactive device adapt to the actual intensity of the user's emotions.

[0050] Optionally, if the dialogue emotion recognition result does not meet the preset acceptance criteria, the user emotion result is determined based on the auxiliary recognition result, including: If the dialogue content corresponding to the dialogue content data is a short sentence with incomplete semantics, the user emotion result is supplemented and determined by combining the auxiliary recognition result, or the previous valid user emotion result is retained. If the dialogue content data is not obtained, a temporary judgment result is determined based on the auxiliary recognition result, and feedback with an intensity lower than a preset intensity is output based on the temporary judgment result; after new dialogue content data is obtained, the user emotion result is updated based on the dialogue emotion recognition result corresponding to the new dialogue content data.

[0051] In this application, when the dialogue emotion recognition result does not meet the preset acceptance conditions, the user's emotion result is determined based on the auxiliary recognition result, specifically including two sub-scenarios. In the first sub-scenarios, the dialogue content data corresponds to a short sentence with incomplete semantics, such as: "um," "oh," "okay," "humph," etc., which cannot be used to determine the emotion. In this case, the auxiliary recognition result is combined to supplement the determination of the user's emotion result, or the previously valid user emotion result is retained. For example, if the user only says "humph," and the auxiliary recognition result points to anger, the processor supplements the determination of the user's emotion result as anger.

[0052] In the second seed scenario of this application, no dialogue content data is obtained, that is, the user does not speak. At this time, a temporary judgment result is determined based on the auxiliary recognition result, and feedback with an intensity lower than the preset intensity is output based on the temporary judgment result, such as lowering the volume, stopping the current playback content and providing mild reassurance, rather than outputting strong expressions such as large movements or high-volume speech. After obtaining new dialogue content data, the user's emotion result is updated based on the dialogue emotion recognition result corresponding to the new dialogue content data.

[0053] This embodiment sets up supplementary judgment and temporary judgment mechanisms for two scenarios: ambiguous dialogue and no dialogue. This enables the interactive device to make an appropriate response when there is insufficient dialogue information, and to correct the judgment in a timely manner after obtaining dialogue information, so as to avoid overreaction or no response due to insufficient information.

[0054] Optionally, the auxiliary expression data includes facial image data, which comprises multiple frames of facial images acquired consecutively within a preset time period; the auxiliary expression data is then used to identify the auxiliary recognition result, including: Emotion recognition is performed on each frame of the multi-frame face images to obtain multiple single recognition results; If the single recognition results obtained consecutively reach a preset number all belong to the same preset facial emotion category, the same preset facial emotion category will be used as the preliminary facial emotion recognition result. If any of the following face images in the multi-frame face images is too small, blurry, occluded, or has its head turned beyond a preset angle, then the face image containing at least one of these conditions is an invalid image, and the invalid image is not used to update the preliminary face emotion recognition result.

[0055] In this application, the auxiliary expression data includes facial image data, which includes multiple frames of facial images captured consecutively within a preset duration. The preset duration can be set to one second, and the number of captures can be set to at least three times.

[0056] Figure 3 This is a flowchart illustrating the facial emotion recognition process provided by the present invention, as shown below. Figure 3 As shown, the processor performs emotion recognition on each frame of face images in multiple frames, and obtains multiple single recognition results. If the single recognition results obtained consecutively reach a preset number of times all belong to the same preset face emotion category, the same preset face emotion category is taken as the preliminary face emotion recognition result. The preset number of times is, for example, three.

[0057] Emotion recognition for each frame of facial images can be performed using a trained facial emotion recognition model. The model includes a feature extraction layer and a classification layer. The feature extraction layer consists of several convolutional and pooling layers, which are used to extract expression features from the normalized facial images. The classification layer consists of fully connected layers, which output the probability that the facial image belongs to each preset facial emotion category. The category with the highest probability is taken as the single recognition result.

[0058] More specifically, the model training process includes: collecting sample images of facial expressions labeled with emotion categories, inputting the sample images into the initial model to obtain the predicted category, updating the model parameters based on the cross-entropy loss between the predicted category and the labeled category, and iteratively training until the loss converges.

[0059] If a face image in a multi-frame face image contains at least one of the following: it is too small, blurry, occluded, or has its head turned beyond a preset angle, the face image containing at least one of the above conditions is an invalid image and will not be used to update the preliminary facial emotion recognition result.

[0060] For example, if a child user turns their head or covers their face with their hand during the data collection process, the corresponding frame is judged as an invalid image, and the processor retains the previous preliminary facial emotion recognition result unchanged.

[0061] This embodiment uses a mechanism that requires multiple consecutive frames to form a preliminary result, combined with the removal of invalid images, to reduce misjudgments caused by brief facial expressions, head turns, occlusions, and single-frame images, thereby improving the reliability of facial emotion recognition results as an auxiliary basis.

[0062] Optionally, the method further includes: If the preliminary facial emotion recognition result meets the secondary confirmation conditions, the preliminary facial emotion recognition result is temporarily suspended, and a question to confirm the emotion is output to the user, or facial emotion recognition continues within a preset observation period; wherein, the secondary confirmation conditions include at least one of the following: different preset facial emotion categories change repeatedly within the preset period, preset easily confused emotion categories appear alternately, the preliminary facial emotion recognition result is inconsistent with the dialogue content data, or the collected facial image is an invalid image; If the user's response indicates their own emotion, or if the single recognition results obtained in the continued recognition process reach a preset number of times and belong to the same preset facial emotion category and do not conflict with the dialogue content data, the confirmed preset facial emotion category will be used as the auxiliary recognition result. If the user does not respond to their own emotion within the preset confirmation time and the recognition continues without obtaining a single recognition result belonging to the same preset facial emotion category, the previous valid user emotion result will be retained, or a neutral response will be output.

[0063] In this application, if the preliminary facial emotion recognition result meets the secondary confirmation condition, the processor temporarily suspends the use of the preliminary facial emotion recognition result and outputs a query statement to the user to confirm the emotion, or continues to perform facial emotion recognition within a preset observation period, which can be set to three to five seconds.

[0064] The secondary confirmation conditions include at least one of the following: different preset facial emotion categories change repeatedly within a preset time period; preset easily confused emotion categories appear alternately, such as surprise and fear, which are similar in facial features and easily alternate; the preliminary facial emotion recognition result is inconsistent with the dialogue content data, such as the preliminary facial emotion recognition result being happy while the user says "I'm angry"; invalid images exist in multiple frames of facial images. The question used to confirm the emotion is selected in a form that is easy for the user to answer, such as "Are you surprised, or a little scared?"

[0065] If the user's response indicates their own emotion, or if the single recognition results obtained consecutively up to a preset number of times during continued recognition belong to the same preset facial emotion category and do not conflict with the dialogue content data, the processor will use the confirmed preset facial emotion category as the auxiliary recognition result; for example, if the recognition result changes repeatedly between surprise and fear, and the user says "I am afraid", the confirmed category is fear.

[0066] If the user does not respond with their own emotion within the preset confirmation time and the recognition continues without obtaining a single recognition result belonging to the same preset facial emotion category, the processor will retain the last valid user emotion result or output a neutral response. A neutral response is a response that does not carry obvious emotional tendencies and whose movement amplitude and volume are at a normal level.

[0067] This embodiment sets up a secondary confirmation mechanism for unstable and easily confused face recognition results, and maintains the existing result or outputs a neutral response when the confirmation fails, thus avoiding making erroneous strong responses based on unreliable face recognition results.

[0068] Optionally, the dialogue content data includes user voice data or user-input text data; performing emotion recognition on the dialogue content data to obtain dialogue emotion recognition results includes: The user's voice data is converted into text to obtain the current dialogue text, or the text data input by the user is used as the current dialogue text; Emotion recognition is performed by combining the current dialogue text with the content of at least one previous round of historical dialogue to obtain the dialogue emotion recognition result.

[0069] In this application, the dialogue content data includes user voice data or user-inputted text data.

[0070] Figure 4 This is a flowchart illustrating the dialogue emotion recognition process provided by the present invention, as shown below. Figure 4 As shown, the processor converts user voice data into text through speech recognition to obtain the current dialogue text, or uses the text data input by the user as the current dialogue text; then, it combines the current dialogue text with the content of at least one previous round of historical dialogue to perform emotion recognition and obtain the dialogue emotion recognition result. The number of rounds of historical dialogue content can be set to one to three previous rounds.

[0071] Emotion recognition based on historical dialogue content can be achieved using a trained text emotion classification model. The model includes an embedding layer, an encoding layer, and a classification layer. The historical dialogue content and the current dialogue text are concatenated in chronological order to form an input sequence. The input sequence is converted into a vector representation by the embedding layer, and contextual semantic features are extracted by the encoding layer. The classification layer outputs the probability distribution of emotion categories, and the category with the highest probability is taken as the dialogue emotion recognition result.

[0072] More specifically, the model training process includes: collecting multi-turn dialogue data with emotion category labels in the context of child companionship, using the spliced ​​dialogue sequence as training input, iteratively updating the model parameters based on the cross-entropy loss between the model's predicted category and the labeled category, until the recognition accuracy on the validation set no longer improves.

[0073] This embodiment combines the current dialogue text with historical dialogue content as the basis for recognition, enabling emotion recognition to utilize contextual information from previous and subsequent dialogues, thus reducing misjudgments caused by relying solely on a single sentence.

[0074] Optionally, emotion recognition is performed by combining the current dialogue text with the content of at least one previous round of historical dialogue to obtain the dialogue emotion recognition result, including: When multiple emotions appear simultaneously in the current dialogue text, and the current dialogue text contains the user's expression of their own emotions, the emotion corresponding to the user's expression of their own emotions is taken as the dialogue emotion recognition result. If multiple emotions appear simultaneously in the current dialogue text, and there is no expression of the user's own emotions in the current dialogue text, the emotion corresponding to the main purpose expressed at the end of the current dialogue text shall be taken as the dialogue emotion recognition result. If the emotion cannot be determined based on the current dialogue text, the dialogue emotion recognition result is determined to be in a pending confirmation state; wherein, the dialogue emotion recognition result in the pending confirmation state does not meet the preset acceptance conditions.

[0075] In this application, when multiple emotions appear simultaneously in the current dialogue text and the current dialogue text contains the user's expression of their own emotions, the processor takes the emotion corresponding to the user's expression of their own emotions as the dialogue emotion recognition result; for example, if the user says "I'm a little scared, can you stay with me?", which contains an expression of their own fear, the dialogue emotion recognition result is fear.

[0076] When multiple emotions appear simultaneously in the current dialogue text, and there is no expression of the user's own emotions in the current dialogue text, the emotion corresponding to the main purpose expressed at the end of the current dialogue text is taken as the dialogue emotion recognition result.

[0077] If the emotion cannot be determined based on the current dialogue text, the processor determines the dialogue emotion recognition result to be in a pending confirmation state. The dialogue emotion recognition result in the pending confirmation state does not meet the preset acceptance conditions. At this time, a simple question can be asked to the user to obtain further information. The pending confirmation state only indicates that the emotion cannot be determined at the moment and does not constitute a new emotion category.

[0078] This embodiment sets up selection and fallback rules for scenarios where multiple emotions coexist in a sentence or where the emotion is uncertain, making the determination process of dialogue emotion recognition results deterministic and avoiding the fluctuation of judgment caused by the mutual interference of multiple emotions.

[0079] Optionally, the negative expression is an expression that contains a negative component for an emotion word, the question expression is an expression that asks the interactive device or a third party about the emotion, and the paraphrase expression is a paraphrase of the emotion in the story content, lyrics, or third-party discourse.

[0080] In this application, a negative statement refers to a statement that contains a negative element in an emotion word, such as "I am not angry" or "I am not afraid." The emotion word is in a negative context, and the user's emotion cannot be judged based on the emotion word itself. A question statement refers to a statement that asks an interactive device or a third party about emotions, such as "Are you angry?" or "Is he sad?" The emotions in the question do not belong to the user's own emotions. A paraphrase statement refers to a paraphrase of the emotions in the story content, lyrics, or third-party statements, such as "The little rabbit in the story is very scared." The emotions of the object being paraphrased cannot be directly taken as the user's emotions.

[0081] In this application, the identification of the three types of expressions can be achieved through semantic analysis methods such as negative word detection, interrogative sentence detection, and quotation structure detection, or the text sentiment classification model can jointly output the expression type while identifying the sentiment.

[0082] This embodiment defines the meaning and identification method of the three expressions in the preset acceptance conditions, so that the acceptance judgment has an operable execution basis, and eliminates the situation where words with literal emotional words that are inconsistent with the user's true emotions are directly accepted from the source.

[0083] Optionally, the auxiliary expression data includes action command data, which is action data of the user acting on the interactive device detected by the sensor; recognizing the auxiliary expression data to obtain auxiliary recognition results includes: Based on the severity, number of times, and duration of the action command data, the action command data is divided into ordinary interaction signals or strong action signals, which serve as the auxiliary recognition result. Neither the ordinary interactive signal nor the stronger action signal independently determines the user's emotional outcome.

[0084] In one embodiment, the auxiliary expression data includes motion command data, which is the action data of the user acting on the interactive device detected by the sensor. The sensor may include a tactile sensor set on the forehead of the robot and an acceleration sensor for detecting shaking. The detected actions include, for example, patting the device's forehead or shaking the device.

[0085] The processor classifies the action command data into ordinary interactive signals or stronger action signals based on the severity, number of actions, and duration of the action, as an auxiliary recognition result. For example, tapping the forehead once within two seconds is classified as an ordinary interactive signal, which is regarded as a call or ordinary interaction. Tapping the forehead twice or more in succession within two seconds, or shaking the device continuously for more than one second, is classified as a stronger action signal. The above time and number thresholds can be adjusted according to the actual situation of the product.

[0086] Neither ordinary interactive signals nor strong action signals determine the user's emotional outcome on their own. That is, action command data only participates in the judgment when combined with dialogue content data or other auxiliary expression data. For example, if a user taps the device on the forehead while saying "Stay with me," the dialogue emotion recognition result meets the preset acceptance conditions. The tapping action does not change the judgment conclusion and is only used to enhance the close interaction.

[0087] This embodiment classifies action command data according to severity, frequency, and duration, and limits action signals from determining emotion on their own. This avoids misjudging ordinary interactive actions as negative emotions and also prevents a single action from overshadowing a clear dialogue meaning.

[0088] Optionally, the dialogue emotion recognition result is one of a plurality of preset dialogue emotion categories, which include: happy, angry, afraid, sad, coquettish, confused, curious, and gentle; The preset facial emotion categories include happy, sad, angry, surprised, afraid, disgusted, and normal.

[0089] In one embodiment, the dialogue emotion recognition result is one of several preset dialogue emotion categories, including happy, angry, afraid, sad, coquettish, confused, curious, and gentle. Coquettishness refers to a user using soft language to request companionship or closeness; confusion refers to a user seeking explanation due to misunderstanding; curiosity refers to a user wanting to explore new information even after understanding the basic content; and gentleness refers to a user comforting or encouraging the other person. The preset facial emotion categories include happy, sad, angry, surprised, afraid, disgusted, and normal, consistent with the output categories of the facial emotion recognition model.

[0090] The processor saves the dialogue emotion recognition results and the preliminary facial emotion recognition results separately. The two results have different category systems. The processor saves the results separately to avoid forcibly converting facial categories such as surprise, disgust or normal into dialogue categories such as coquettishness or confusion. The two results are used by the comprehensive judgment and feedback intensity determination stages respectively.

[0091] This embodiment limits the two recognition results to their respective fixed category sets and saves them separately, ensuring the consistency between the recognition results and the output of the recognition model, and avoiding semantic distortion caused by forced conversion across category systems.

[0092] Optionally, based on the anthropomorphic emotion, determine the device expression data, including: Based on the pre-established mapping relationship between anthropomorphic emotions and expression methods, the expression method corresponding to the anthropomorphic emotion is determined as the device expression data; In the mapping relationship, the anthropomorphic emotions are divided into multiple intensity levels according to their intensity, and different intensity levels correspond to different expression methods. The expression methods include at least one of facial expression, body movement expression, voice tone expression, and light expression. The higher the intensity of the anthropomorphic emotion, the more obvious the body movement expression.

[0093] In this application, the processor determines the expression mode corresponding to the anthropomorphic emotion based on a pre-established mapping relationship between anthropomorphic emotions and expression modes, and uses it as device expression data.

[0094] Figure 5 This is a schematic diagram illustrating the mapping relationship between anthropomorphic emotions and expression methods provided by the present invention, such as... Figure 5 As shown, the anthropomorphic emotions in the mapping relationship are divided into multiple intensity levels according to the intensity of the emotion. For example, strong emotions include happiness, anger, fear and sadness; moderate intensity emotions include coquetry, confusion and curiosity; and mild emotions include gentleness, apology and seriousness. Among them, apology and seriousness are the response attitudes output by the device.

[0095] The modes of expression include at least one of facial expressions, body language, tone of voice, and lighting. The higher the intensity of the anthropomorphic emotion, the more obvious the body language. For example, when the anthropomorphic emotion is happiness, the pet robot will rapidly cross and swing its ears while making small movements. When the anthropomorphic emotion is confusion, the pet robot will adopt a posture with one ear erect and the other drooping. When the anthropomorphic emotion is gentle, it will only use a soft tone of voice, soft lighting, and slow ear swaying as subtle expressions.

[0096] This embodiment determines the expression method by mapping the intensity of emotions, so that strong emotions are expressed through obvious and perceptible actions, while mild emotions are presented through tone of voice and micro-movements. This helps child users intuitively understand the emotional state of the interactive device and enhances the emotional transmission effect during companionship.

[0097] Optionally, the data expressed by the device includes at least one of the following: When the user's emotional state is anger, the device's expression data includes lowering the volume, stopping the currently playing content, and expressing an apology; When the user's emotional outcome is fear or sadness, the device's expressive data includes comforting content output in a reassuring tone; In the case where the user's emotional outcome is confusion, the device expression data includes prior output content that has been reinterpreted in a simplified manner; When the user's emotional outcome is curiosity, the device's expressive data includes answers to the user's questions and extended content.

[0098] When the user's emotional outcome is anger, the device's data expression includes lowering the volume, stopping the currently playing content, and expressing an apology; when the user's emotional outcome is fear or sadness, the device's data expression includes comforting content delivered in a reassuring tone; when the user's emotional outcome is confusion, the device's data expression includes reinterpreting the previously output content in a simplified manner; and when the user's emotional outcome is curiosity, the device's data expression includes answers to the user's questions and extended content.

[0099] For example, if a child user says "I didn't understand," the user's emotional result is confusion, and the pet robot will re-explain the content in a simpler way.

[0100] This embodiment sets corresponding feedback content for different user emotional outcomes, so that the response of the interactive device is further aligned with the user's actual needs on the basis of emotion matching, thereby improving the pertinence and quality of the interaction.

[0101] The emotion expression device provided by the present invention is described below. The emotion expression device described below and the emotion expression method described above can be referred to in correspondence.

[0102] Figure 6 This is a schematic diagram of the structure of the emotion expression device provided by the present invention, as shown below. Figure 6 As shown, it includes: The acquisition module 610 is used to acquire user expression data; wherein, the user expression data includes dialogue content data and auxiliary expression data, and the auxiliary expression data is data in the user expression data other than the dialogue content data, which is used to represent the user's emotions; The recognition module 620 is used to perform emotion recognition on the dialogue content data to obtain the dialogue emotion recognition result, and to recognize the auxiliary expression data to obtain the auxiliary recognition result; The determining module 630 is configured to: determine the dialogue emotion recognition result as the user's emotion result when the dialogue emotion recognition result meets the preset acceptance conditions and the emotion pointed to by the auxiliary recognition result is consistent with the emotion pointed to by the dialogue emotion recognition result; output a confirmation query to the user when the dialogue emotion recognition result meets the preset acceptance conditions and the emotion pointed to by the auxiliary recognition result is inconsistent with the emotion pointed to by the dialogue emotion recognition result, and maintain the dialogue emotion recognition result as the user's emotion result or redetermine the user's emotion result based on the user's response to the confirmation query; and determine the user's emotion result based on the auxiliary recognition result when the dialogue emotion recognition result does not meet the preset acceptance conditions. The preset acceptance conditions include: the dialogue content data contains a user's expression of their own emotions, and the expression is not a negative expression, a question expression, or a paraphrase expression. The expression module 640 is used to determine the anthropomorphic emotion of the interactive device based on the user emotion result, determine device expression data based on the anthropomorphic emotion, and output the device expression data; wherein, the device expression data is used to characterize the anthropomorphic emotion of the interactive device in response to the user expression data.

[0103] Figure 7 This is a structural schematic diagram of the pet robot provided by the present invention, as shown below. Figure 7 As shown, the pet robot may include: a processor 710, a communications interface 720, a memory 730, and a communication bus 740. The processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute an emotion expression method. This method includes: acquiring user expression data; wherein the user expression data includes dialogue content data and auxiliary expression data, and the auxiliary expression data is data in the user expression data other than the dialogue content data used to characterize the user's emotions. Emotion recognition is performed on the dialogue content data to obtain the dialogue emotion recognition result, and the auxiliary expression data is recognized to obtain the auxiliary recognition result; If the dialogue emotion recognition result meets the preset acceptance conditions, and the emotion pointed to by the auxiliary recognition result is consistent with the emotion pointed to by the dialogue emotion recognition result, the dialogue emotion recognition result is determined as the user's emotion result; wherein, the preset acceptance conditions include: the dialogue content data contains the user's expression of their own emotions, and the expression is not a negative expression, a question expression, or a paraphrase expression. If the dialogue emotion recognition result meets the preset acceptance conditions, and the emotion pointed to by the auxiliary recognition result is inconsistent with the emotion pointed to by the dialogue emotion recognition result, a confirmation question is output to the user, and based on the user's response to the confirmation question, the dialogue emotion recognition result is maintained as the user's emotion result, or the user's emotion result is redefined; before the response is obtained, the dialogue emotion recognition result is used as the user's emotion result. If the dialogue emotion recognition result does not meet the preset acceptance conditions, the user's emotion result is determined based on the auxiliary recognition result; Based on the user's emotion result, the anthropomorphic emotion of the interactive device is determined, and based on the anthropomorphic emotion, device expression data is determined and output; wherein, the device expression data is used to characterize the anthropomorphic emotion of the interactive device in response to the user's expression data.

[0104] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0105] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, the computer program being executed by a processor, the computer being able to execute the emotion expression method provided by the above methods, the method including: acquiring user expression data; wherein, the user expression data includes dialogue content data and auxiliary expression data, the auxiliary expression data being data used to characterize user emotions in the user expression data other than the dialogue content data; Emotion recognition is performed on the dialogue content data to obtain the dialogue emotion recognition result, and the auxiliary expression data is recognized to obtain the auxiliary recognition result; If the dialogue emotion recognition result meets the preset acceptance conditions, and the emotion pointed to by the auxiliary recognition result is consistent with the emotion pointed to by the dialogue emotion recognition result, the dialogue emotion recognition result is determined as the user's emotion result; wherein, the preset acceptance conditions include: the dialogue content data contains the user's expression of their own emotions, and the expression is not a negative expression, a question expression, or a paraphrase expression. If the dialogue emotion recognition result meets the preset acceptance conditions, and the emotion pointed to by the auxiliary recognition result is inconsistent with the emotion pointed to by the dialogue emotion recognition result, a confirmation question is output to the user, and based on the user's response to the confirmation question, the dialogue emotion recognition result is maintained as the user's emotion result, or the user's emotion result is redefined; before the response is obtained, the dialogue emotion recognition result is used as the user's emotion result. If the dialogue emotion recognition result does not meet the preset acceptance conditions, the user's emotion result is determined based on the auxiliary recognition result; Based on the user's emotion result, the anthropomorphic emotion of the interactive device is determined, and based on the anthropomorphic emotion, device expression data is determined and output; wherein, the device expression data is used to characterize the anthropomorphic emotion of the interactive device in response to the user's expression data.

[0106] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the emotion expression method provided by the above methods, the method comprising: acquiring user expression data; wherein the user expression data includes dialogue content data and auxiliary expression data, the auxiliary expression data being data in the user expression data other than the dialogue content data used to characterize user emotions; Emotion recognition is performed on the dialogue content data to obtain the dialogue emotion recognition result, and the auxiliary expression data is recognized to obtain the auxiliary recognition result; If the dialogue emotion recognition result meets the preset acceptance conditions, and the emotion pointed to by the auxiliary recognition result is consistent with the emotion pointed to by the dialogue emotion recognition result, the dialogue emotion recognition result is determined as the user's emotion result; wherein, the preset acceptance conditions include: the dialogue content data contains the user's expression of their own emotions, and the expression is not a negative expression, a question expression, or a paraphrase expression. If the dialogue emotion recognition result meets the preset acceptance conditions, and the emotion pointed to by the auxiliary recognition result is inconsistent with the emotion pointed to by the dialogue emotion recognition result, a confirmation question is output to the user, and based on the user's response to the confirmation question, the dialogue emotion recognition result is maintained as the user's emotion result, or the user's emotion result is redefined; before the response is obtained, the dialogue emotion recognition result is used as the user's emotion result. If the dialogue emotion recognition result does not meet the preset acceptance conditions, the user's emotion result is determined based on the auxiliary recognition result; Based on the user's emotion result, the anthropomorphic emotion of the interactive device is determined, and based on the anthropomorphic emotion, device expression data is determined and output; wherein, the device expression data is used to characterize the anthropomorphic emotion of the interactive device in response to the user's expression data.

[0107] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0108] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method of expressing emotion, characterized by, Applied to interactive devices, the method includes: Acquire user expression data; wherein, the user expression data includes dialogue content data and auxiliary expression data, and the auxiliary expression data is data used to characterize user emotions in the user expression data other than the dialogue content data; Emotion recognition is performed on the dialogue content data to obtain the dialogue emotion recognition result, and the auxiliary expression data is recognized to obtain the auxiliary recognition result; If the dialogue emotion recognition result meets the preset acceptance conditions, and the emotion pointed to by the auxiliary recognition result is consistent with the emotion pointed to by the dialogue emotion recognition result, the dialogue emotion recognition result is determined as the user's emotion result; wherein, the preset acceptance conditions include: the dialogue content data contains the user's expression of their own emotions, and the expression is not a negative expression, a question expression, or a paraphrase expression. If the dialogue emotion recognition result meets the preset acceptance conditions, and the emotion pointed to by the auxiliary recognition result is inconsistent with the emotion pointed to by the dialogue emotion recognition result, a confirmation question is output to the user, and based on the user's response to the confirmation question, the dialogue emotion recognition result is maintained as the user's emotion result, or the user's emotion result is redefined; before the response is obtained, the dialogue emotion recognition result is used as the user's emotion result. If the dialogue emotion recognition result does not meet the preset acceptance conditions, the user's emotion result is determined based on the auxiliary recognition result; Based on the user's emotion result, the anthropomorphic emotion of the interactive device is determined, and based on the anthropomorphic emotion, device expression data is determined and output; wherein, the device expression data is used to characterize the anthropomorphic emotion of the interactive device in response to the user's expression data.

2. The method for expressing emotion according to claim 1, characterized in that, The method further includes: If the emotion indicated by the auxiliary recognition result is consistent with the emotion indicated by the dialogue emotion recognition result, the feedback intensity corresponding to the device expression data is determined based on the obviousness of the emotion expression in the auxiliary recognition result. The more pronounced the emotional expression, the stronger the feedback.

3. The method for expressing emotion according to claim 1, characterized in that, If the dialogue emotion recognition result does not meet the preset acceptance conditions, the user emotion result is determined based on the auxiliary recognition result, including: If the dialogue content corresponding to the dialogue content data is a short sentence with incomplete semantics, the user emotion result is supplemented and determined by combining the auxiliary recognition result, or the previous valid user emotion result is retained. If the dialogue content data is not obtained, a temporary judgment result is determined based on the auxiliary recognition result, and feedback with an intensity lower than a preset intensity is output based on the temporary judgment result; after new dialogue content data is obtained, the user emotion result is updated based on the dialogue emotion recognition result corresponding to the new dialogue content data.

4. The method for expressing emotion according to claim 1, characterized in that, The auxiliary expression data includes facial image data, which includes multiple frames of facial images collected consecutively within a preset time period; The auxiliary expression data is identified to obtain auxiliary identification results, including: Emotion recognition is performed on each frame of the multi-frame face images to obtain multiple single recognition results; If the single recognition results obtained consecutively reach a preset number all belong to the same preset facial emotion category, the same preset facial emotion category will be used as the preliminary facial emotion recognition result. If any of the following conditions are present in the multi-frame face images: the face image is blurred, occluded, or the head is turned beyond a preset angle, then the face image containing at least one of these conditions is considered invalid and will not be used to update the preliminary facial emotion recognition result.

5. The method for expressing emotion according to claim 4, characterized in that, The method further includes: If the preliminary facial emotion recognition result meets the secondary confirmation conditions, the preliminary facial emotion recognition result is temporarily suspended, and a question to confirm the emotion is output to the user, or facial emotion recognition continues within a preset observation period; wherein, the secondary confirmation conditions include at least one of the following: different preset facial emotion categories change repeatedly within the preset period, preset easily confused emotion categories appear alternately, the preliminary facial emotion recognition result is inconsistent with the dialogue content data, or the collected facial image is an invalid image; If the user's response indicates their own emotion, or if the single recognition results obtained in the continued recognition process reach a preset number of times and belong to the same preset facial emotion category and do not conflict with the dialogue content data, the confirmed preset facial emotion category will be used as the auxiliary recognition result. If the user does not respond to their own emotion within the preset confirmation time and the recognition continues without obtaining a single recognition result belonging to the same preset facial emotion category, the previous valid user emotion result will be retained, or a neutral response will be output.

6. The method for expressing emotion according to claim 1, characterized in that, The dialogue content data includes user voice data or user-input text data; emotion recognition is performed on the dialogue content data to obtain dialogue emotion recognition results, including: The user's voice data is converted into text to obtain the current dialogue text, or the text data input by the user is used as the current dialogue text; Emotion recognition is performed by combining the current dialogue text with the content of at least one previous round of historical dialogue to obtain the dialogue emotion recognition result.

7. The method for expressing emotion according to claim 6, characterized in that, Emotion recognition is performed by combining the current dialogue text with the content of at least one previous round of historical dialogue to obtain the dialogue emotion recognition result, including: When multiple emotions appear simultaneously in the current dialogue text, and the current dialogue text contains the user's expression of their own emotions, the emotion corresponding to the user's expression of their own emotions is taken as the dialogue emotion recognition result. If multiple emotions appear simultaneously in the current dialogue text, and there is no expression of the user's own emotions in the current dialogue text, the emotion corresponding to the main purpose expressed at the end of the current dialogue text shall be taken as the dialogue emotion recognition result. If the emotion cannot be determined based on the current dialogue text, the dialogue emotion recognition result is determined to be in a pending confirmation state; wherein, the dialogue emotion recognition result in the pending confirmation state does not meet the preset acceptance conditions.

8. The method for expressing emotion according to claim 1, characterized in that, The negation statement refers to a statement that contains a negative component in the emotion word; the question statement refers to a statement that asks the interactive device or a third party about the emotion; and the paraphrase statement refers to a paraphrase of the emotion in the story content, lyrics, or third-party discourse.

9. The method for expressing emotion according to claim 1, characterized in that, The auxiliary expression data includes action command data, which is action data of the user acting on the interactive device detected by the sensor; The auxiliary expression data is identified to obtain auxiliary identification results, including: Based on the severity, number of times, and duration of the action command data, the action command data is divided into ordinary interaction signals or strong action signals, which serve as the auxiliary recognition result. Neither the ordinary interactive signal nor the stronger action signal independently determines the user's emotional outcome.

10. The method for expressing emotion according to claim 4, characterized in that, The dialogue emotion recognition result is one of multiple preset dialogue emotion categories, which include: happy, angry, afraid, sad, coquettish, confused, curious, and gentle. The preset facial emotion categories include happy, sad, angry, surprised, afraid, disgusted, and normal.

11. The method for expressing emotion according to claim 1, characterized in that, Based on the anthropomorphic emotions, the device's expression data is determined, including: Based on the pre-established mapping relationship between anthropomorphic emotions and expression methods, the expression method corresponding to the anthropomorphic emotion is determined as the device expression data; In the mapping relationship, the anthropomorphic emotions are divided into multiple intensity levels according to their intensity, and different intensity levels correspond to different expression methods. The expression methods include at least one of facial expression, body movement expression, voice tone expression, and light expression. The higher the intensity of the anthropomorphic emotion, the more obvious the body movement expression.

12. The method for expressing emotion according to claim 1, characterized in that, The device expresses data including at least one of the following: When the user's emotional state is anger, the device's expression data includes lowering the volume, stopping the currently playing content, and expressing an apology; When the user's emotional outcome is fear or sadness, the device's expressive data includes comforting content output in a reassuring tone; In the case where the user's emotional outcome is confusion, the device expression data includes prior output content that has been reinterpreted in a simplified manner; When the user's emotional outcome is curiosity, the device's expressive data includes answers to the user's questions and extended content.

13. An emotion expression device, characterized in that, The device includes: An acquisition module is used to acquire user expression data; wherein, the user expression data includes dialogue content data and auxiliary expression data, and the auxiliary expression data is data used to characterize user emotions in the user expression data other than the dialogue content data; The recognition module is used to perform emotion recognition on the dialogue content data to obtain the dialogue emotion recognition result, and to recognize the auxiliary expression data to obtain the auxiliary recognition result; The determination module is configured to: determine the dialogue emotion recognition result as the user's emotion result when the dialogue emotion recognition result meets preset acceptance conditions and the emotion pointed to by the auxiliary recognition result is consistent with the emotion pointed to by the dialogue emotion recognition result; output a confirmation query to the user when the dialogue emotion recognition result meets the preset acceptance conditions and the emotion pointed to by the auxiliary recognition result is inconsistent with the emotion pointed to by the dialogue emotion recognition result, and maintain the dialogue emotion recognition result as the user's emotion result or redetermine the user's emotion result based on the user's response to the confirmation query; and determine the user's emotion result based on the auxiliary recognition result when the dialogue emotion recognition result does not meet the preset acceptance conditions. The preset acceptance conditions include: the dialogue content data contains a user's expression of their own emotions, and the expression is not a negative expression, a question expression, or a paraphrase expression. An expression module is used to determine the anthropomorphic emotion of the interactive device based on the user's emotion result, determine device expression data based on the anthropomorphic emotion, and output the device expression data; wherein, the device expression data is used to characterize the anthropomorphic emotion of the interactive device in response to the user's expression data.

14. A pet robot, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the emotion expression method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the emotion expression method as described in any one of claims 1 to 12.

Images