Micro-expression generation method, device, medium and program product for an expression head

Abstract

The application discloses a micro-expression generation method and device of an expression head, a medium and a program product, and relates to the field of human-computer interaction. The micro-expression generation method of the expression head comprises the following steps: collecting facial original analysis data of a user when the user interacts with the expression head and audio data of the user; determining a smiling clue value and a smiling response scene category based on the facial original analysis data and the audio data of the user; calculating an expression head smiling proportion according to a smiling proportion upper limit value, a smiling proportion lower limit value, a basic smiling proportion and the smiling clue value of the smiling response scene category; calculating an expression head facial organ traction amplitude based on the expression head smiling proportion, the smiling response scene category and an expression head facial organ traction amplitude threshold value, and generating an expression head smiling traction track according to the expression head facial organ traction amplitude. The technical scheme of the embodiment of the application can improve the performance effect of the smiling expression of the expression head of the humanoid robot and greatly improve the user experience.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to a method, device, medium, and program product for generating micro-expressions in facial expressions. Background Technology

[0002] Humanoid robots typically need to convey friendliness, responsiveness, encouragement, or polite feedback to users through facial expressions. However, current robot expression output methods mostly rely on fixed expression templates or single-action playback. Smiling expressions often only have two states: "smiling" and "not smiling," resulting in stiff expressions, limited variety, and inconsistency with the current interaction scenario.

[0003] In real-world communication, a user's smile may have different meanings. For example, a user's smile when greeting someone, a smile after being encouraged, a smile when expressing gratitude, or a polite smile in an awkward situation will all require a different range of facial expressions, eye involvement, and mouth corner traction when responding to them. If a humanoid robot's facial expression head directly imitates a user's expression, it is prone to over-exaggeration, delayed response, or inappropriate expressions. If the humanoid robot's facial expression head only plays preset smile templates, it is difficult to adjust the softness of the smile and the coordination of facial features according to the interactive content. Summary of the Invention

[0004] This invention provides a method, device, medium, and program product for generating micro-expressions in facial expressions, in order to solve the problem that existing humanoid robot facial expressions have poor smiling expression effects and poor user experience.

[0005] According to one aspect of the present invention, a method for generating micro-expressions in an emoji head is provided, comprising:

[0006] Collect raw facial analysis data and user audio data when users interact with emoji avatars;

[0007] Based on raw facial analysis data and user audio data, we determine smile cue values ​​and smile response scenario categories.

[0008] The smile percentage of the facial expression head is calculated based on the upper limit, lower limit, basic smile percentage, and smile cue value of the smile response scenario category.

[0009] Based on the proportion of smiling faces, the type of smiling response scenario, and the threshold of facial organ traction amplitude, the facial organ traction amplitude is calculated, and the facial smiling traction trajectory is generated according to the facial organ traction amplitude.

[0010] According to another aspect of the present invention, a micro-expression generation device for facial expressions is provided, comprising:

[0011] The data acquisition module is used to collect raw facial analysis data and user audio data when users interact with the emoji avatars.

[0012] The data preprocessing module is used to determine the smile cue value and smile response scenario category based on the raw facial analysis data and user audio data;

[0013] The facial expression smile percentage calculation module is used to calculate the facial expression smile percentage based on the upper limit, lower limit, basic smile percentage, and smile cue value of the smile response scenario category.

[0014] The facial expression smile traction trajectory generation module is used to calculate the traction amplitude of facial organs based on the proportion of facial expressions smiling, the type of smiling response scenario, and the threshold of traction amplitude of facial organs, and to generate the facial expression smile traction trajectory based on the traction amplitude of facial organs.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] At least one processor; and a memory communicatively connected to said at least one processor;

[0017] The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the micro-expression generation method for facial expressions as described in any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the micro-expression generation method for facial expression heads according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the micro-expression generation method for facial expression heads according to any embodiment of the present invention.

[0020] The technical solution of this invention collects raw facial analysis data and user audio data when the user interacts with the emoji head. Based on the raw facial analysis data and user audio data, it determines the smile cue value and smile response scenario category. Then, based on the upper limit value, lower limit value, basic smile ratio, and smile cue value of the smile response scenario category, it calculates the smile ratio of the emoji head. Furthermore, based on the smile ratio of the emoji head, the smile response scenario category, and the facial organ traction amplitude threshold of the emoji head, it calculates the facial organ traction amplitude of the emoji head and generates the smile traction trajectory of the emoji head based on the facial organ traction amplitude. This solution generates responsive smiles based on smile cue values ​​and interactive scenarios. It controls the overall strength of the expression by the proportion of the smile and the naturalness of the smile by the traction amplitude of facial organs. This ensures that the final smile traction trajectory is suitable for the scenario and avoids the stiffness caused by a single template. Furthermore, the generation of the smile traction trajectory does not rely on EEG signals, emotion trend prediction, or robot outer coating material deformation models, resulting in low implementation costs. It is suitable for deployment in service robots, companion robots, explanation robots, and interactive display smile heads. This solution solves the problem of poor smile expression performance and user experience in existing humanoid robot smile heads, and can improve the performance of smile expressions in humanoid robot smile heads, greatly enhancing the user experience.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 The flowchart shows a method for generating micro-expressions in an emoji head according to Embodiment 1 of the present invention.

[0024] Figure 2 This is a flowchart of a method for generating micro-expressions in an emoji head according to Embodiment 2 of the present invention;

[0025] Figure 3 This is a schematic diagram of the micro-expression generation logic of an emoji head provided in Embodiment 3 of the present invention;

[0026] Figure 4 This is a schematic diagram of the structure of a micro-expression generation device for an emoji head provided in Embodiment 4 of the present invention;

[0027] Figure 5 A schematic diagram of an electronic device that can be used to implement embodiments of the present invention is shown. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first," "second," etc., in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] Example 1

[0031] Figure 1 This is a flowchart illustrating a method for generating micro-expressions from an emoji in Embodiment 1 of the present invention. This embodiment is applicable to situations where an emoji dynamically generates a soft and harmonious smiling expression. The method can be executed by a micro-expression generation device for the emoji, which can be implemented in hardware and / or software. This device can be configured in an electronic device and can be a virtual device executing the micro-expression generation method. The electronic device may include an emoji, etc. Figure 1 As shown, the method includes:

[0032] Step 110: Collect raw facial analysis data and user audio data when the user interacts with the emoji avatar.

[0033] The raw facial analysis data can be user facial images captured by the facial expression camera. This raw facial analysis data can include user facial image frames or short fragments of user facial images. It can be used to detect the position of the user's mouth corners, the degree of lip opening and closing, the degree of eye opening and closing, facial orientation, and the relative position between the user and the facial expression camera. The user audio data can be the user's audio data during interaction with the facial expression camera. This user audio data can be used to determine whether the user is engaging in interactive behaviors such as greeting, thanking, responding, or taking a photo. For example, the user audio data can include the user's speech, laughter, pauses in speech, and changes in the volume of the user's speech.

[0034] In this embodiment of the invention, raw facial analysis data and user audio data can be collected by the emoji head when the user interacts with the emoji head.

[0035] Step 120: Based on the original facial analysis data and user audio data, determine the smile cue value and the smile response scenario category.

[0036] The smile cue value can be a quantitative indicator of the facial organ contraction amplitude when the facial expression head smiles. The smile response scenario category can be the type of scenario in which the facial expression head responds to a smiling expression. For example, smile response scenario categories can include greeting smiles, thank-you smiles, encouraging smiles, polite smiles, and photo-taking smiles. Smile response scenario categories can be identified using smile response scenario codes. The smile response scenario category determination does not output categories such as "happy," "negative," and "neutral," but rather outputs a scenario code directly related to the robot's response behavior.

[0037] For example, a greeting smile is triggered when a user approaches the emoticon and a greeting, slight smile, or brief pause occurs, outputting a low to medium smile percentage, making the emoticon appear friendly but not overly enthusiastic. A thank-you smile is triggered after a user expresses gratitude or completes a service process, outputting a medium smile percentage and enhancing the corners of the mouth and cheeks for a gentler response. An encouraging smile is triggered when a user completes a task, answers a question, or needs positive feedback, outputting a medium to high smile percentage and increasing eye involvement for a more encouraging expression. A polite smile is triggered when the user's expression is not obvious but a polite response is required for the interaction, outputting a low smile percentage to avoid exaggerated expressions. A photo-taking smile is triggered when a user requests a photo, a group photo, or looks directly at the camera, outputting a high but limited smile percentage to ensure the emoticon displays a stable photo-taking smile.

[0038] In this embodiment of the invention, relevant calculation data can be extracted from the original face analysis data and user audio data according to the needs of smile cue value calculation, so as to calculate the smile cue value, and the original face analysis data and user audio data can be analyzed based on the preset smile response scenario category discrimination conditions to obtain the smile response scenario category.

[0039] Step 130: Calculate the smile percentage of the facial expression based on the upper limit, lower limit, basic smile percentage, and smile cue value of the smile response scenario category.

[0040] The upper limit of the smile percentage can be a pre-set maximum smile percentage corresponding to a specific smile response scenario category. The lower limit of the smile percentage can be a pre-set minimum smile percentage corresponding to the same scenario category. The base smile percentage can be the base smile percentage assigned to the smile response scenario category. The final smile percentage of the facial expression can be the output smile percentage of the facial expression. The smile percentage can be understood as the ratio of the output smile expression to the preset standard smile expression; that is, the average ratio of the facial feature stretching amplitude of the output smile expression to the stretching amplitude of the facial features in the preset standard smile expression. The value range of the smile percentage is 0-1.

[0041] In this embodiment of the invention, the smile ratio to be restricted can be calculated based on the smile cue value of the basic smile ratio. Then, based on the upper limit value and the lower limit value of the smile ratio, the smile ratio to be restricted is clamped to obtain the smile ratio of the facial expression. It is not a simple copy of the user's smile amplitude, but a suitable responsive smile ratio is generated according to the interaction scenario.

[0042] Step 140: Based on the proportion of smiling faces, the type of smiling response scenario, and the threshold of facial organ traction amplitude, calculate the facial organ traction amplitude and generate the facial smiling traction trajectory according to the facial organ traction amplitude.

[0043] The threshold for the traction amplitude of facial organs can be a threshold value for the traction amplitude of facial organs. This threshold can include an upper limit threshold and a lower limit threshold for the traction amplitude of facial organs. The traction amplitude of facial organs can be the degree of deformation of the facial organs. The traction trajectory of a smiling expression can be the continuous trajectory of the smiling expression output by the facial head.

[0044] In this embodiment of the invention, a threshold value for the traction amplitude of facial organs corresponding to the smile response scenario category can be determined first. Then, based on the proportion of the smiling face and the smile response scenario category, a limit value for the traction amplitude of the facial organs can be determined. Then, based on the threshold value for the traction amplitude of the facial organs, the limit value for the traction amplitude of the facial organs can be clamped to obtain the traction amplitude of the facial organs. Then, according to the traction amplitude of the facial organs, a smile traction trajectory of the facial face is generated to control the facial organs of the facial face to display a smile according to the smile traction trajectory of the facial face.

[0045] The technical solution of this invention collects raw facial analysis data and user audio data when the user interacts with the emoji head. Based on the raw facial analysis data and user audio data, it determines the smile cue value and smile response scenario category. Then, based on the upper limit value, lower limit value, basic smile ratio, and smile cue value of the smile response scenario category, it calculates the smile ratio of the emoji head. Furthermore, based on the smile ratio of the emoji head, the smile response scenario category, and the facial organ traction amplitude threshold of the emoji head, it calculates the facial organ traction amplitude of the emoji head and generates the smile traction trajectory of the emoji head based on the facial organ traction amplitude. This solution generates responsive smiles based on smile cue values ​​and interactive scenarios. It controls the overall strength of the expression by the proportion of the smile and the naturalness of the smile by the traction amplitude of facial organs. This ensures that the final smile traction trajectory is suitable for the scenario and avoids the stiffness caused by a single template. Furthermore, the generation of the smile traction trajectory does not rely on EEG signals, emotion trend prediction, or robot outer coating material deformation models, resulting in low implementation costs. It is suitable for deployment in service robots, companion robots, explanation robots, and interactive display smile heads. This solution solves the problem of poor smile expression performance and user experience in existing humanoid robot smile heads, and can improve the performance of smile expressions in humanoid robot smile heads, greatly enhancing the user experience.

[0046] Example 2

[0047] Figure 2 This is a flowchart of a micro-expression generation method for facial expressions provided in Embodiment 2 of the present invention. This embodiment is a specific embodiment based on the above embodiment, and provides specific optional implementation methods for updating the facial expression organ traction amplitude after calculating the facial expression organ traction amplitude based on the facial expression organ's smile ratio, smile response scenario category, and facial expression organ traction amplitude threshold. Figure 2 As shown, the method includes:

[0048] Step 210: Collect raw facial analysis data and user audio data when the user interacts with the emoji avatar.

[0049] Step 220: Based on the original facial analysis data and user audio data, determine the smile cue value and the smile response scenario category.

[0050] In an optional embodiment of the present invention, the smile cue value may include a visual smile cue value and a voice interaction cue value; determining the smile cue value and the smile response scenario category based on the original face analysis data and the user audio data may include: calculating the visual smile cue value based on the original face analysis data, and calculating the voice interaction cue value based on the user audio data; determining the smile response scenario category based on the visual smile cue value, the voice interaction cue value, and the interactive text data in the user audio data.

[0051] The visual smile cue value can be a smile cue value calculated based on raw facial analysis data. The audio interaction cue value can be a smile cue value calculated based on user audio data. The interactive text data can be the dialogue text between the user and the emoticon in the user audio data. For example, the emoticon's speech recognition function or dialogue system can identify the interactive text data from the user's audio data. The interactive text data can include, but is not limited to, phrases such as "Hello," "Thank you," "Can I take a picture?" and "Well done," to help determine the category of the smile response scenario.

[0052] In this embodiment of the invention, the positions of the user's mouth corner, the middle of the user's lips, the eyelids, and the corners of the eyes can be extracted from the original facial analysis data. Based on these positions, the degree of change in the user's facial change area can be determined, and a visual smile cue value can be obtained. Audio pauses, laughter, and credibility can be extracted from the user's audio data, and a sound interaction cue value can be calculated. The visual smile cue value, the sound interaction cue value, and the interactive text data in the user's audio data can be combined according to preset smile response scenario type discrimination conditions to determine the smile response scenario category.

[0053] In an optional embodiment of the present invention, calculating the visual smile cue value based on raw facial analysis data may include: calculating the visual smile cue value based on the following formula:

[0054] ;

[0055] in, This represents the upturn value of the corners of the mouth in the raw facial analysis data; This represents the eye involvement value in the raw face analysis data; Indicates the credibility of face images in the raw face analysis data; This indicates the weight of the upturned corners of the mouth value; Indicates the weight of the eye-related participation value; Indicates the credibility weight of a face image; Represents the amplitude limiting function; This represents the visual smile cue value. The clamp function can limit the output to between 0 and 1.

[0056] The parameters are as follows: Mouth corner upturn value (corresponding to the degree of upward curve of the mouth corners); Eye involvement value (corresponding to the distance between the upper and lower eyelids and the degree of change in eye corner texture); Face image credibility (the degree of credibility of the face image); Mouth corner upturn value weight (corresponding to the weight of the mouth corner upturn value); Eye involvement value weight (corresponding to the weight of the eye involvement value); Face image credibility weight (corresponding to the weight of the face image credibility). The weights for mouth corner upturn value, eye involvement value, and face image credibility need to be preset.

[0057] In this embodiment of the invention, corresponding upturn values ​​for different upturned corners of the mouth can be set in advance; corresponding eye participation values ​​can be set in advance for different combinations of the distance between the upper and lower eyelids and the degree of change in the eye corner texture; and facial image credibility can be set in advance for different combinations of user facial orientation and the proportion of the visible area of ​​the user's face.

[0058] In an optional embodiment of the present invention, calculating the voice interaction cue value based on user audio data may include: calculating the voice interaction cue value based on the following formula:

[0059] ;

[0060] in, Indicates the audio laughter cue value; Indicates the audio pause cue value; Indicates the credibility of the audio; Indicates the weight of the laughter cue; Indicates the weight of paused clues; Indicates the credibility weight of the audio; Indicates the value of voice interaction cues. This represents the amplitude limiting function.

[0061] The audio laughter cue value describes whether laughter (such as a soft laugh or whining) is present when the user speaks; the higher the decibel level, the closer the audio laughter cue value is to 1. The audio pause cue value describes whether there are pauses when the user speaks; the longer the pause, the closer the audio pause cue value is to 1. Audio reliability describes the reliability of the user's audio data. Audio reliability can be determined based on audio signal clarity, noise ratio, volume, background noise, and audio integrity; the higher the audio reliability, the closer it is to 1. Laughter cue weights can be weights corresponding to the audio laughter cue values. Pause cue weights can be weights corresponding to the audio pause cue values. Audio reliability weights can be weights corresponding to the audio reliability. The laughter cue weights, pause cue weights, and audio reliability weights need to be preset.

[0062] Step 230: Calculate the smile percentage of the facial expression based on the upper limit, lower limit, basic smile percentage, and smile cue value of the smile response scenario category.

[0063] In an optional embodiment of the present invention, calculating the smile percentage of the facial expression based on the upper limit of the smile percentage, the lower limit of the smile percentage, the basic smile percentage, and the smile cue value of the smile response scenario category may include: calculating the smile percentage of the facial expression based on the following formula:

[0064] ;

[0065] in, This indicates the percentage of smiling faces in the facial expression; Indicates the type of scenario for responding with a smile; Indicates the value of voice interaction cues; Indicates the visual smile cue value; This represents the smile inhibition value; This indicates the percentage of basic smiles; This indicates the lower limit of the percentage of smiles in a particular scenario. This indicates the upper limit of the percentage of smiles in a particular scenario. This indicates that the smile has the highest weighting. This indicates that the smile has the second highest weighting. This indicates that smiling accounts for the third largest weighting; The smile is the fourth most important factor; T represents the scene enhancement value.

[0066] The smile suppression value can be a preset suppression value for a smiling expression. The scene enhancement value can be used to represent the enhancement value of the interactive text data or dialogue corresponding to the scene. The smile suppression value and scene enhancement value can be determined by technical personnel. The first weight of the smile proportion can be the weight corresponding to the visual smile cue value. The second weight of the smile proportion can be the weight corresponding to the audio interaction cue value. The third weight of the smile proportion can be the weight corresponding to the scene enhancement value. The fourth weight of the smile proportion can be the weight corresponding to the smile suppression value. The first, second, third, and fourth weights of the smile proportion are preset.

[0067] For example, in a greeting smile scenario, Smax(c) can be set to a lower value to avoid excessive expression upon initial contact; in an encouraging smile scenario, Smax(c) can be appropriately increased to allow the facial expression head to output a more obvious encouraging smile; in a polite smile scenario, even if the smile cue value is high, the smile is kept restrained by a lower upper limit.

[0068] Step 240: Calculate the traction amplitude of facial organs based on the proportion of smiling faces, the category of smiling response scenarios, and the threshold of traction amplitude of facial organs.

[0069] In an optional embodiment of the present invention, calculating the traction amplitude of facial organs based on the proportion of smiling facial expressions, the category of smiling response scenarios, and the threshold of traction amplitude of facial organs may include: calculating the traction amplitude of facial organs based on the following formula:

[0070] ;

[0071] in, This represents the target traction amplitude of the i-th facial organ in the expression head, i.e., the maximum expected traction amplitude of the i-th facial organ in the expression head; This indicates the percentage of smiling faces in the facial expression; This represents the configuration proportion of the i-th facial feature in the expression head that matches the scene type of the smiling response; This represents the preset traction scale for the i-th facial organ in the expression head; This represents the neutral position compensation value for the i-th facial feature in the expression head; This represents the lower limit threshold of the traction amplitude for the i-th facial organ in the expression head; This represents the upper limit threshold of the traction amplitude of the i-th facial organ in the expression head.

[0072] The neutral position compensation value can be a pre-configured neutral emotion compensation value. Neutral emotion can be understood as an emotion between happiness and negativity. The preset traction scale can be used to characterize the traction range of facial organs. The configuration proportions of facial organs can be understood as the traction stretching proportions of facial organs. The traction range of facial organs in the expression head does not require establishing a stress-strain model of the expression head's outer covering material, nor does it require converting the expression amplitude into biomimetic muscle duty cycle parameters; instead, the target traction amplitude is directly generated in the organ traction coordinate system of the expression head.

[0073] For example, the traction amplitude of facial organs can include, but is not limited to, the traction amplitude of the left corner of the mouth, the traction amplitude of the right corner of the mouth, the traction amplitude of the upper or lower lip contour, the traction amplitude of the left eyelid, the traction amplitude of the right eyelid, the traction amplitude of the left cheek, the traction amplitude of the right cheek, the traction amplitude of the left eyebrow, and the traction amplitude of the right eyebrow.

[0074] A greeting smile features a higher proportion of mouth corner traction, a lower proportion of eye traction, and a moderate proportion of cheek traction, resulting in a gentle expression. A thank-you smile maintains symmetrical mouth corner traction and a slightly higher cheek traction, making the expression appear mild. An encouraging smile appropriately increases the traction proportions of the mouth corners, cheeks, and eyes, giving the expression a more positive feedback. A polite smile has a limited overall traction range, with slight traction at the corners of the mouth and only weak involvement from the eyes and eyebrows. A photo-taking smile features a higher proportion of mouth corner traction, a moderate proportion of eye traction, and maintains stability for a longer period.

[0075] Step 250: Obtain the organ traction amplitude constraints and update the facial organ traction amplitude according to the organ traction amplitude constraints.

[0076] The organ traction amplitude constraint can be a condition that restricts the traction amplitude of facial organs. For example, the organ traction amplitude constraint can include left-right symmetry constraint, eye-mouth coordination constraint, speech avoidance constraint, scene upper limit constraint, and initial pose constraint. Specifically, the left-right symmetry constraint requires that when the user is facing the facial expression head, the difference in traction amplitude between the left and right corners of the mouth must not exceed a preset threshold. The eye-mouth coordination constraint requires that when the corner of the mouth traction amplitude exceeds a certain threshold, the eye traction amplitude must at least reach the corresponding minimum participation value to avoid a stiff expression where only the mouth smiles without any eye movement. The speech avoidance constraint requires that when the facial expression head is broadcasting a speech, the lip traction amplitude is reduced to the allowable range for broadcasting, prioritizing clear lip movements. The scene upper limit constraint requires that different smile response scenarios have different maximum traction ranges to prevent polite smiles from being amplified into exaggerated smiles. The initial pose constraint requires that if the current facial expression head is in the fallback phase of the previous expression, the new smile traction amplitude is calculated starting from the current organ position to avoid abrupt changes.

[0077] In this embodiment of the invention, organ traction amplitude constraints can be obtained, and then the traction amplitude of facial organs can be determined to meet the requirements based on the organ traction amplitude constraints. When the traction amplitude of facial organs does not meet the organ traction amplitude constraints, the traction amplitude of facial organs is updated based on the organ traction amplitude constraints.

[0078] Step 260: Generate the facial expression head traction trajectory based on the updated facial organ traction amplitude.

[0079] In this embodiment of the invention, a smiling traction trajectory of the facial expression head can be generated based on the updated traction amplitude of the facial organs.

[0080] For example, the traction amplitude of facial organs in facial expressions is converted into a continuous output trajectory. Each smiling expression includes at least three stages:

[0081] 1) Initial stage: Facial organs gradually move from their current position to the corresponding traction range to form a natural smile.

[0082] 2) Holding phase: The facial organs are held near the corresponding traction amplitude for a certain period of time to complete the facial expression response.

[0083] 3) Regression phase: The facial organs gradually return to a neutral position or the initial position of the next expression from the corresponding traction amplitude.

[0084] Optionally, the initial smile phase can be shorter than the decline phase, allowing the smile to appear more promptly and fade more gently. The duration of the smile's duration is determined by the type of situation in which the smile is being responded to; for example, a polite smile lasts shorter, while a smile for a photo lasts longer.

[0085] The trajectory of a smiling face can be represented as: Where Pi(t) represents the traction amplitude of the i-th facial organ of the facial expression head at time t, Pstart(i) represents the initial traction position of the i-th facial organ of the facial expression head, and Fstage(t) represents the normalized progress function of the current stage, characterizing the speed at which the facial organ of the facial expression head moves from the initial position to the target position. Fstage(t) is usually a value between 0 and 1. stage represents the current stage, such as the starting stage, the holding stage, or the falling stage. This represents the target traction amplitude of the i-th facial organ in the expression head. The starting position of the facial organ can be extracted from the expression head state data. The expression head state data can include whether the expression head is currently speaking, the current initial position of the facial organ, and whether the previous round of expression output has ended, etc., to avoid forcibly switching expressions at inappropriate times to output a smile. The changes in the facial organ traction amplitude frames of the expression head can be seen in Table 1:

[0086] Table 1. Reference Table of Changes in Facial Organ Traction Amplitude Frames in Expression Heads

[0087]

[0088] The technical solution of this invention collects raw facial analysis data and user audio data when a user interacts with an emoji. Based on the raw facial analysis data and user audio data, it determines the smile cue value and smile response scenario category. Then, based on the upper limit value, lower limit value, basic smile proportion, and smile cue value of the smile response scenario category, it calculates the smile proportion of the emoji. Based on the smile proportion of the emoji, the smile response scenario category, and the emoji facial organ traction amplitude threshold, it calculates the emoji facial organ traction amplitude. It further obtains organ traction amplitude constraints and updates the emoji facial organ traction amplitude according to the organ traction amplitude constraints. Based on the updated emoji facial organ traction amplitude, it generates the emoji smile traction trajectory. This solution generates responsive smiles based on smile cue values ​​and interactive scenarios. It controls the overall strength of the expression by the proportion of the smile and the naturalness of the smile by the traction amplitude of facial organs. This ensures that the final smile traction trajectory is suitable for the scenario and avoids the stiffness caused by a single template. Furthermore, the generation of the smile traction trajectory does not rely on EEG signals, emotion trend prediction, or robot outer coating material deformation models, resulting in low implementation costs. It is suitable for deployment in service robots, companion robots, explanation robots, and interactive display smile heads. This solution solves the problem of poor smile expression performance and user experience in existing humanoid robot smile heads, and can improve the performance of smile expressions in humanoid robot smile heads, greatly enhancing the user experience.

[0089] Example 3

[0090] Embodiment 3 of the present invention provides an optional embodiment of a method for generating micro-expressions in facial expressions, the specific implementation of which can be found in the following embodiments. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here.

[0091] In this approach, raw facial analysis data and user audio data are not used to generate cross-modal emotion matching vectors or to predict the user's emotional trend in the next moment. Instead, they are used to extract smile trigger cues and response scenario cues from the current interaction segment.

[0092] In a specific example, such as Figure 3 As shown, the micro-expression generation logic of the emoji is as follows:

[0093] The system collects raw facial analysis data and user audio data. Visual smile cue values ​​and audio interaction cue values ​​are extracted from these data, and then the smile response scenario category is determined based on these values ​​and the interactive text data. The suitability of outputting a smile is determined based on the smile response expression category. If the smile response expression category is outside a preset range, it indicates that outputting a smile is not suitable, and a neutral expression is maintained, or the expression reverts to a safe expression. When the smile response expression category is within a preset range, it indicates that outputting a smile is suitable. The proportion of smiling faces is calculated, and based on this proportion and other relevant data, the traction amplitude of facial organs is calculated. If the traction amplitude of facial organs meets the organ traction amplitude constraint (i.e., coordination constraint), a smile traction trajectory is generated, and the smiling expression is output. If the organ traction amplitude constraint is not met, the traction amplitude of facial organs is updated, and it is determined whether the coordination constraint is satisfied.

[0094] Compared with existing fixed-template robot facial expression output methods, this invention has at least the following advantages: it can generate responsive smiles based on the user's current smiling cues and interaction scenario, avoiding the stiffness caused by a single template; it controls the overall strength of the expression by controlling the proportion of the smile, so that the robot does not simply copy the user's expression, but outputs a smile suitable for the scenario; it calculates the traction amplitude of each organ to ensure coordination between the corners of the mouth, eyelids, cheeks, and eyebrows, improving the naturalness of the smiling expression; it generates a traction trajectory through three stages: starting, holding, and falling back, making the appearance and disappearance of the smiling expression smoother; and it does not rely on EEG signals, emotion trend prediction, or robot outer coating material deformation models, resulting in lower system implementation costs and making it suitable for deployment in service robots, companion robots, explanation robots, and interactive display devices.

[0095] Example 4

[0096] like Figure 4 As shown, this embodiment of the invention also provides a micro-expression generation device for facial expressions, the device comprising:

[0097] Data acquisition module 310 is used to collect raw facial analysis data and user audio data when the user interacts with the emoji head;

[0098] The data preprocessing module 320 is used to determine the smile cue value and smile response scenario category based on the original face analysis data and user audio data;

[0099] The facial expression smile percentage calculation module 330 is used to calculate the facial expression smile percentage based on the upper limit value, lower limit value, basic smile percentage, and smile cue value of the smile response scenario category.

[0100] The facial expression head smile traction trajectory generation module 340 is used to calculate the traction amplitude of facial organs based on the proportion of facial expression head smiles, the category of smile response scenarios, and the threshold of facial organ traction amplitude, and to generate the facial expression head smile traction trajectory based on the traction amplitude of facial organs.

[0101] The technical solution of this invention collects raw facial analysis data and user audio data when the user interacts with the emoji head. Based on the raw facial analysis data and user audio data, it determines the smile cue value and smile response scenario category. Then, based on the upper limit value, lower limit value, basic smile ratio, and smile cue value of the smile response scenario category, it calculates the smile ratio of the emoji head. Furthermore, based on the smile ratio of the emoji head, the smile response scenario category, and the facial organ traction amplitude threshold of the emoji head, it calculates the facial organ traction amplitude of the emoji head and generates the smile traction trajectory of the emoji head based on the facial organ traction amplitude. This solution generates responsive smiles based on smile cue values ​​and interactive scenarios. It controls the overall strength of the expression by the proportion of the smile and the naturalness of the smile by the traction amplitude of facial organs. This ensures that the final smile traction trajectory is suitable for the scenario and avoids the stiffness caused by a single template. Furthermore, the generation of the smile traction trajectory does not rely on EEG signals, emotion trend prediction, or robot outer coating material deformation models, resulting in low implementation costs. It is suitable for deployment in service robots, companion robots, explanation robots, and interactive display smile heads. This solution solves the problem of poor smile expression performance and user experience in existing humanoid robot smile heads, and can improve the performance of smile expressions in humanoid robot smile heads, greatly enhancing the user experience.

[0102] Optionally, the data preprocessing module 320 is used to determine the smile cue value and smile response scenario category based on the original face analysis data and the user audio data, including: calculating the visual smile cue value based on the original face analysis data, and calculating the voice interaction cue value based on the user audio data; and determining the smile response scenario category based on the visual smile cue value, the voice interaction cue value, and the interactive text data in the user audio data.

[0103] Optionally, the data preprocessing module 320 is used to calculate the visual smile cue value based on the following formula:

[0104] ;

[0105] in, This represents the upturn value of the corners of the mouth in the raw facial analysis data; This represents the eye involvement value in the raw face analysis data; This indicates the credibility of the facial images in the raw facial analysis data; This indicates the weight of the upturned corners of the mouth value; Indicates the weight of the eye-related participation value; Indicates the credibility weight of a face image; Represents the amplitude limiting function; This represents the visual smile cue value.

[0106] Optionally, the data preprocessing module 320 is used to calculate the voice interaction cue value based on the following formula:

[0107] ;

[0108] in, Indicates the audio laughter cue value; Indicates the audio pause cue value; Indicates the credibility of the audio; Indicates the weight of the laughter cue; Indicates the weight of paused clues; Indicates the credibility weight of the audio; This represents the value of the voice interaction cue.

[0109] Optionally, the facial expression smile ratio calculation module 330 is used to calculate the facial expression smile ratio based on the following formula:

[0110] ;

[0111] in, This indicates the percentage of the facial expression head that is smiling; Indicates the type of scenario for responding with a smile; This represents the voice interaction cue value. This represents the smile inhibition value; This indicates the percentage of the base smile; This represents the lower limit of the percentage of smiles in the aforementioned smile response scenario category; This represents the upper limit of the percentage of smiles in the aforementioned smile response scenario category; This indicates that the smile has the highest weighting. This indicates that the smile has the second highest weighting. This indicates that smiling accounts for the third largest weighting; The smile is the fourth most important factor; T represents the scene enhancement value.

[0112] Optionally, the facial expression head traction trajectory generation module 340 is used to calculate the traction amplitude of the facial organs based on the following formula: ;

[0113] in, This represents the target traction amplitude of the i-th facial organ in the expression head; This indicates the percentage of the facial expression head that is smiling; This represents the configuration proportion of the i-th facial feature in the expression head that matches the scene type of the smiling response; This represents the preset traction scale for the i-th facial organ in the expression head; This represents the neutral position compensation value for the i-th facial feature in the expression head; This represents the lower limit threshold of the traction amplitude for the i-th facial organ in the expression head; This represents the upper limit threshold of the traction amplitude of the i-th facial organ in the expression head.

[0114] Optionally, the micro-expression generation device for facial expressions also includes a facial organ traction amplitude update module, used to obtain organ traction amplitude constraints; wherein, the organ traction amplitude constraints include at least one of the following: left and right symmetry constraints of the corners of the mouth, eye-mouth coordination constraints, speaking avoidance constraints, and initial smiling posture constraints; and the facial organ traction amplitude is updated according to the organ traction amplitude constraints.

[0115] Example 5

[0116] Figure 5 A schematic diagram of an electronic device that can be used to implement embodiments of the present invention is shown. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the invention described and / or claimed herein.

[0117] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as ROM 12, RAM 13, etc., communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from the storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An I / O interface 15 is also connected to the bus 14. The ROM 12 is a read-only memory, the RAM 13 is a random access memory, and the I / O interface 15 is an input / output interface.

[0118] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0119] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the micro-expression generation method for facial expressions.

[0120] In some embodiments, the method for generating micro-expressions of facial expressions can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the method for generating micro-expressions of facial expressions described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the method for generating micro-expressions of facial expressions by any other suitable means (e.g., by means of firmware).

[0121] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0122] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0123] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0124] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0125] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0126] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS servers, such as high management difficulty and weak business scalability.

[0127] This application also discloses a computer program product, which includes a computer program that, when executed by a processor, implements the micro-expression generation method for facial expressions provided in any embodiment of this application. This program product and the micro-expression generation method for facial expressions disclosed in the embodiments of this application belong to the same inventive concept, and therefore will not be described in detail here.

[0128] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0129] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for generating micro-expressions in an emoji, characterized in that, include: Collect raw facial analysis data and user audio data when users interact with emoji avatars; Based on the raw facial analysis data and the user audio data, determine the smile cue value and the smile response scenario category; The smile percentage of the facial expression head is calculated based on the upper limit of the smile percentage, the lower limit of the smile percentage, the basic smile percentage, and the smile cue value of the smile response scenario category. Based on the proportion of smiling faces, the type of smiling response scenario, and the threshold of facial organ traction amplitude, the facial organ traction amplitude is calculated, and a smiling trajectory is generated according to the facial organ traction amplitude.

2. The method according to claim 1, characterized in that, The smile cue values ​​include visual smile cue values ​​and vocal interaction cue values; Based on the raw facial analysis data and the user audio data, smile cue values ​​and smile response scenario categories are determined, including: Based on the original facial analysis data, the visual smile cue value is calculated, and based on the user audio data, the voice interaction cue value is calculated. The smile response scenario category is determined based on the visual smile cue value, the audio interaction cue value, and the interactive text data in the user audio data.

3. The method according to claim 2, characterized in that, Based on the raw facial analysis data, the visual smile cue value is calculated, including: The visual smile cue value is calculated based on the following formula: ； in, This represents the upturn value of the corners of the mouth in the raw facial analysis data; This represents the eye involvement value in the raw face analysis data; This indicates the credibility of the facial images in the raw facial analysis data; This indicates the weight of the upturned corners of the mouth value; Indicates the weight of the eye involvement value; Indicates the credibility weight of a face image; Represents the amplitude limiting function; This represents the visual smile cue value.

4. The method according to claim 2, characterized in that, Based on the user audio data, the sound interaction cue value is calculated, including: The voice interaction cue value is calculated based on the following formula: ； in, Indicates the audio laughter cue value; Indicates the audio pause cue value; Indicates the credibility of the audio; Indicates the weight of the laughter cue; Indicates the weight of paused clues; Indicates the credibility weight of the audio; This represents the value of the voice interaction cue.

5. The method according to claim 3, characterized in that, The smile percentage of the facial expression is calculated based on the upper limit, lower limit, and basic smile percentage of the smile response scenario category, as well as the smile cue value, including: The percentage of smiling facial expressions is calculated based on the following formula: ； in, This indicates the percentage of the facial expression head that is smiling; Indicates the type of scenario for responding with a smile; This represents the voice interaction cue value. This represents the smile inhibition value; This indicates the percentage of the base smile; This represents the lower limit of the percentage of smiles in the aforementioned smile response scenario category; This represents the upper limit of the percentage of smiles in the aforementioned smile response scenario category; This indicates that the smile has the highest weighting. This indicates that the smile has the second highest weighting. This indicates that smiling accounts for the third largest weighting; The smile is the fourth most important factor; T represents the scene enhancement value.

6. The method according to claim 1, characterized in that, Based on the percentage of smiling faces, the category of smiling response scenarios, and the threshold for the traction amplitude of facial organs, the traction amplitude of facial organs is calculated, including: The traction amplitude of the facial organs is calculated based on the following formula: ； in, This represents the target traction amplitude of the i-th facial organ in the expression head; This indicates the percentage of the facial expression head that is smiling; This represents the configuration proportion of the i-th facial feature in the expression head that matches the scene type of the smiling response; This represents the preset traction scale for the i-th facial organ in the expression head; This represents the neutral position compensation value for the i-th facial feature in the expression head; This represents the lower limit threshold of the traction amplitude for the i-th facial organ in the expression head; This represents the upper limit threshold of the traction amplitude of the i-th facial organ in the expression head.

7. The method according to claim 1, characterized in that, After calculating the facial organ traction amplitude based on the proportion of smiling faces, the type of smiling response scenario, and the threshold for facial organ traction amplitude, the process further includes: Obtain organ traction amplitude constraints; wherein, the organ traction amplitude constraints include at least one of the following: left and right corner symmetry constraint, eye-mouth coordination constraint, speaking avoidance constraint, and initial smiling posture constraint. Update the traction amplitude of the facial organs according to the organ traction amplitude constraints.

8. An electronic device, characterized in that, The electronic device includes: At least one processor, and a memory communicatively connected to said at least one processor; The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the micro-expression generation method for the facial expression head according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the micro-expression generation method of the facial expression head according to any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the micro-expression generation method for the facial expression head according to any one of claims 1-7.

Images