Eye expression generation method and device, equipment and storage medium

Abstract

The application relates to the field of computer vision, and provides an eye expression generation method, device, equipment and storage medium. The method comprises the following steps: based on the eye gaze position obtained from the previous state set of a target object in a previous frame and a real-time driving set of a current frame, performing contour shape adjustment on an initial eye expression of the target object in the current frame; and based on dynamic behavior data and a preset data priority, performing eye expression adjustment on the obtained candidate eye expression to obtain a target eye expression of the target object in the current frame. By adjusting the contour shape and superimposing the dynamic behavior data, the shape dislocation or time sequence dislocation is fundamentally eliminated, the eye expression image with high coordination between the basic shape and the dynamic behavior is generated in real time, and the eye expression delicacy, naturalness and authenticity of the target object are improved.

Description

Technical Field

[0001] This application relates to the field of computer vision and provides a method, apparatus, device, and storage medium for generating eye expressions. Background Technology

[0002] Artificial intelligence terminals with displays (such as home companion robots, in-vehicle assistants, and public tour guide robots) typically convey emotions and attention to users through digital eyes.

[0003] Previous approaches used offline frame sequence-based methods, which pre-rendered several sets of Portable Network Graphics (PNG) or Graphics Interchange Format (GIF) frames and played them sequentially after an event was triggered. While simple to implement, this approach suffers from low flexibility, high storage costs, and the need to create a separate complete frame sequence for each new expression. It also lacks the ability to link with real-time emotional states, making maintenance and expansion difficult.

[0004] Therefore, the current mainstream approach is based on 2D skeletal skinning, which models eye components (such as the eye socket and pupil) in layers and uses bones to drive their scaling and rotation to achieve facial expression changes. However, this approach has limited deformation capabilities, making it difficult to reproduce realistic eyelid curvature and pupil anomalies. Furthermore, the separation of facial expression shape control from dynamic behavior control can easily result in a mask-like or unnatural appearance.

[0005] Therefore, a new method for generating eye expressions is needed to overcome the aforementioned shortcomings. Summary of the Invention

[0006] This application provides a method, apparatus, device, and storage medium for generating eye expressions, in order to solve the problems of low coordination, insufficient real-time performance, and stiff expressions caused by the separation of static form and dynamic behavior in traditional solutions.

[0007] In a first aspect, embodiments of this application provide a method for generating eye expressions, including: Obtain the preceding state set of the target object in the previous frame and the real-time driving set of the current frame; the preceding state set refers to the historical expression information of the target object's eye expression in the previous frame, and the real-time driving set refers to the real-time expression information used to drive the generation of the target object's eye expression in the current frame. Based on the eye gaze position obtained from the preceding state set and the real-time driving set, the outline shape of the target object's initial eye expression in the current frame is adjusted to obtain the candidate eye expression of the target object in the current frame. Based on dynamic behavior data and preset data priorities, the candidate eye expressions are adjusted to obtain the target eye expression of the target object in the current frame; the dynamic behavior data refers to the data that triggers changes in the target object's eye expression.

[0008] Secondly, embodiments of this application also provide an apparatus for generating eye expression images, comprising: The state acquisition module is used to acquire the previous state set of the target object in the previous frame and the real-time driving set of the current frame; the previous state set refers to the historical expression information of the target object's eye expression in the previous frame, and the real-time driving set refers to the real-time expression information used to drive the generation of the target object's eye expression in the current frame. The shape adjustment module is used to adjust the outline shape of the initial eye expression of the target object in the current frame based on the eye gaze position obtained from the preceding state set and the real-time drive set, so as to obtain the candidate eye expression of the target object in the current frame. The behavior overlay module is used to adjust the eye expressions of the candidate eye expressions based on dynamic behavior data and preset data priorities, so as to obtain the target eye expression of the target object in the current frame; the dynamic behavior data refers to the data that triggers the change of the target object's eye expression.

[0009] Optionally, the shape adjustment module is used for: Extract the eye gaze position of the target object in the previous frame from the preceding state set, and extract the eye gaze position of the target object in the current frame from the real-time drive set; Based on the extracted eye gaze positions of the two frames, the gaze position offset of the target object in the two adjacent frames is obtained; When the gaze position of the target object in two adjacent frames is offset by more than the position offset threshold, the first movement mode is adopted to adjust the outline shape of the initial eye expression of the target object in the current frame and obtain the candidate eye expression of the target object in the current frame. When the gaze position offset of the target object in two adjacent frames is not greater than the position offset threshold, the second movement mode is adopted to adjust the contour shape of the initial eye expression and obtain the candidate eye expression of the target object in the current frame.

[0010] Optionally, the shape adjustment module is used for: Based on the duration of the gaze jump from the current frame to the trigger time of this gaze jump and the gaze position of the target object in the current frame, the positions of the control points of each pupil contour of the target object's initial eye expression in the current frame are adjusted. After confirming that the control points of each pupil contour have been adjusted, based on the jump duration and the gaze position of the target object in the current frame, the position of each orbital contour control point of the target object in the current frame is further adjusted.

[0011] Optionally, the shape adjustment module is used for: Based on the duration of the gaze jump from the current frame to the trigger time of this gaze jump and the gaze position of the target object in the current frame, the positions of each pupil contour control point and each eye socket contour control point of the target object's initial eye expression in the current frame are adjusted.

[0012] Optionally, the behavior overlay module is used for: Based on the facial expression material information obtained from the real-time driving set, the candidate eye expressions are fused to obtain the eye expressions of the target object in the current frame. Based on the emotional state information obtained from the preceding state set and the real-time driving set, the eye expressions of the material are processed by emotion fusion to obtain the emotional eye expressions of the target object in the current frame. Based on the facial expression flag information obtained from the real-time driving set, the emotional eye expression is subjected to behavior superposition processing to obtain the target eye expression of the target object in the current frame.

[0013] Optionally, the behavior overlay module is used for: If the facial expression material information in the real-time drive set is a material playback instruction, the progress update is performed on the material playback progress of the previous frame in the preceding state set. When the material playback progress of the current frame indicates that the current frame is in the material playback stage, the facial expression material of the first frame and the candidate eye expression are processed by facial expression fusion to obtain the material eye expression of the target object in the current frame. If the facial expression material information in the real-time drive set is a material exit instruction, the material exit progress of the previous frame in the preceding state set is updated. When the material exit progress of the current frame indicates that the current frame is in the material exit stage, the last frame facial expression material and the candidate eye expression are processed by facial expression fusion to obtain the material eye expression of the target object in the current frame. If the expression material information in the real-time drive set is a material switching instruction, a new eye expression material is obtained, and the new first frame eye expression material and the candidate eye expression are processed by expression fusion to obtain the target object's material eye expression in the current frame.

[0014] Optionally, the behavior overlay module is used for: If the first frame of the expression material is not in a closed-eye state, then based on the playback progress of the material in the current frame, a first transition weight is obtained to control the transition speed curve, and based on the first transition weight, the first frame of the expression material and the candidate eye expression are fused to generate the target object's eye expression in the current frame. If the first frame of the expression material shows the eyes closed, then the candidate eye expression is replaced with the first frame of the expression material to obtain the target object's eye expression in the current frame.

[0015] Optionally, the behavior overlay module is used for: Based on the material exit progress of the current frame, a second transition weight is obtained to control the transition speed curve; Based on the second transition weight, the last frame's expression material and the candidate eye expressions are fused to generate the target object's eye expression material in the current frame.

[0016] Optionally, the behavior overlay module is further configured to: If the expression material information in the real-time driver set is a material pause playback command, the fusion step is paused, and the outline shape of the candidate eye expression remains unchanged.

[0017] Optionally, the behavior overlay module is used for: Based on the emotion type and emotion intensity obtained from the preceding state set and the real-time driving set, the emotion switching result of the target object in two adjacent frames is obtained. When the emotion switching result of the target object in two adjacent frames indicates that the target object's emotion has changed drastically, the outline shape of the eye expression of the material is adjusted based on the target emotion type and target emotion intensity of the target object in the current frame to obtain the emotional eye expression of the target object in the current frame. When the emotion switching result indicates that the target object's emotion has not changed drastically, the outline shape adjustment of the eye expression of the material driven by blinking behavior is performed to obtain the emotional eye expression of the target object in the current frame.

[0018] Optionally, the behavior overlay module is used for: Obtain multiple sets of contour control point sequences corresponding to the target emotion type and intensity of the target object in the current frame; For two adjacent sets of contour control point sequences, perform the following operations respectively: perform interpolation processing on the two adjacent sets of contour control point sequences to obtain the interpolated control point sequence under the emotional transition state; Based on multiple sets of contour control point sequences and multiple sets of interpolation control point sequences, the contour shape of the eye expression of the material is adjusted multiple times in succession, and the eye expression obtained by the last adjustment is taken as the emotional eye expression of the target object in the current frame.

[0019] Optionally, the behavior overlay module is used for: When it is determined that the current frame is in the blinking phase, obtain the blinking keyframes and adjacent keyframes associated with the current frame during the blinking process; Interpolation processing is performed on the associated blink keyframes and the blink contour control points contained in adjacent keyframes to obtain the interpolation control point sequence in the blink transition state. Based on the interpolation control point sequence during the blink transition, the outline shape of the eye expression in the material is adjusted to obtain the emotional eye expression of the target object in the current frame.

[0020] Optionally, the behavior overlay module is used for: When it is determined that the eye gaze position of the target object changes drastically between two adjacent frames, perform the following operations on the eye expression of each clip of the target object in sequence: Based on the duration of the jump and the total duration of the jump in the current frame from the trigger time of the current gaze jump, the closure weight of each eye socket contour control point in the eye expression of the material is obtained. Based on the closure weight of each contour control point, the position of each eye socket contour control point in the material's eye expression is adjusted to obtain the emotional eye expression of the target object in the current frame.

[0021] Optionally, if the real-time drive set fails to extract the eye gaze position of the target object in the current frame, the state acquisition module cyclically executes the following out-of-focus search steps until the eye gaze position of the target object in the current frame is found or the set search duration is reached: Based on the generated multiple virtual attention parameters, the target object is controlled to move within a preset range of multiple pupil contour control points of the initial eye expression in the current frame; During the movement, multi-dimensional environmental information around the target object is extracted, and based on the extracted multi-dimensional environmental information, the actual attention parameters for the current frame are regenerated. Based on the actual attention parameters of the current frame, the eye gaze position of the target object in the current frame is obtained.

[0022] Optionally, during the out-of-focus search process, the state acquisition module cyclically executes the following pupil movement steps until the eye fixation position of the target object in the current frame is obtained or the set search duration is reached: Taking the target object's eye gaze position in the previous frame as the starting point of loss, the contour control points of the initial eye expression are controlled to slide inertially along the last eye movement direction before loss. The system controls each contour control point to stop moving for a set duration, and adjusts the inertial reset of each contour control point to drive the pupil center point of the target object in the current frame back to the center position of the eye.

[0023] Optionally, the behavior overlay module is used for: If the expression flag information obtained by the real-time drive set is a forced blink or entering a sleep state, the contour shape of the emotional eye expression is adjusted by blink behavior to obtain the target eye expression of the target object in the current frame.

[0024] Optionally, the behavior overlay module is further configured to: Extract the remaining blink cool-down time from the blink cool-down timer in the preceding state set; If the blinking behavior trigger interval between two adjacent frames of the target object is less than the blinking cooldown time, the blinking cooldown mechanism is activated to prevent the superposition of blinking behavior in the current frame. If the blinking behavior trigger interval between two adjacent frames of the target object is not less than the blinking cooldown time, the blinking cooldown mechanism is deactivated, allowing the blinking behavior to be superimposed in the current frame, and the blinking cooldown timer is restarted to start a new round of cooldown timing.

[0025] Optionally, the eye expression generation device further includes a contour construction module. Before acquiring the previous state set of the target object in the previous frame and the real-time driving set of the current frame, the contour construction module performs the following steps to construct the standard eye contour of the target object in the first frame: Construct multiple sets of contour control point sequences, one set of contour control point sequences is used to depict the contour of an eye component; For each set of contour control point sequences in the standard eye contour, a layer-by-layer interpolation step is performed iteratively until the current layer contour control point sequence contains only two contour control points: interpolation processing is performed based on the two adjacent contour control points of the current layer contour control point sequence and the corresponding chord length accumulation sequence to obtain the interpolated control points located between the two adjacent contour control points, and each interpolated control point generated in the current layer is used as the contour control point sequence of the next layer; the chord length accumulation sequence is used to represent the approximate cumulative arc length from each contour control point to the first contour control point in the target control point index table; Based on each set of contour control point sequences and each set of corresponding multi-layer interpolation control points, a smooth curve contour of each eye component is fitted to form the standard eye contour of the target object in the first frame.

[0026] Optionally, the contour construction module performs the following steps to generate the chord length accumulation sequence of the i-th contour control point: A target control point index table for a single-layer contour control point sequence is constructed using a modular loop approach. Based on the geometric distance between the i-th contour control point and the (i-1)-th contour control point in the target control point index table, and the cumulative chord length sequence of the first (i-1) contour control points, the cumulative chord length sequence of the i-th contour control point is generated, where i is a positive integer.

[0027] The contour building module is used for: For each contour control point in the original control point index table, perform the following operations: Take the index value of the i-th contour control point in the original control point index table and perform a modulo operation with the index value of the last contour control point in the table to obtain a first modulo result; Perform another modulo operation with the first modulo result and the index value of the last contour control point in the table to determine a target contour control point that has a mapping relationship with the i-th contour control point; Use the index value of the target contour control point in the original control point index table as the new index value of the i-th contour control point. Based on the new index values ​​of all contour control points, construct the target control point index table.

[0028] Thirdly, embodiments of this application also provide a robot, including a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor performs the steps of any of the above-described eye expression generation methods.

[0029] Fourthly, embodiments of this application also provide a computer-readable storage medium including program code, which, when the program product is run on a computer device, is used to cause the computer device to perform the steps of any of the above-described eye expression generation methods.

[0030] The beneficial effects of this application are as follows: This application provides a method, apparatus, device, and storage medium for generating eye expressions. The method includes: adjusting the contour shape of the initial eye expression of the target object in the current frame based on the eye gaze position obtained from the previous frame's pre-state set and the current frame's real-time drive set; then, adjusting the eye expression of the obtained candidate eye expressions based on dynamic behavior data and preset data priorities to obtain the target eye expression of the target object in the current frame. By adjusting the contour shape and superimposing dynamic behavior data, the method fundamentally eliminates shape misalignment or temporal disconnection, achieving real-time generation of eye expression images with highly coordinated basic shape and dynamic behavior, thereby improving the delicacy, naturalness, and realism of the target object's eye expression.

[0031] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0032] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is an optional schematic diagram of an application scenario in the embodiments of this application; Figure 2A A schematic diagram illustrating the process of generating real-time eye expressions of a target object according to an embodiment of this application; Figure 2B A logical diagram illustrating the real-time generation of eye expressions of a target object, provided in an embodiment of this application; Figure 2C A schematic diagram illustrating the process of constructing the standard eye contour of the target object in the first frame, as provided in an embodiment of this application; Figure 2D A schematic diagram of the standard eye contour of the target object in the first frame, provided for an embodiment of this application; Figure 2E A schematic diagram of the right eye socket constructed based on layer-by-layer interpolation, provided for an embodiment of this application; Figure 2F A logical diagram illustrating the adjustment of eye contour using a time-sharing movement mode, provided for an embodiment of this application; Figure 2G A logical diagram illustrating the adjustment of eye contour using a simultaneous movement mode, provided for an embodiment of this application; Figure 2H A logical diagram illustrating the generation of eye expressions from source material based on source material playback instructions, provided in an embodiment of this application. Figure 2I A logical diagram illustrating the generation of eye expressions from source material based on a source material exit command, as provided in this embodiment of the application. Figure 2J A logical diagram illustrating the generation of eye expressions from source material based on source material switching instructions, as provided in this embodiment of the application; Figure 2K This is a logical diagram illustrating the adjustment of contour shape based on the target emotion type and intensity, as provided in an embodiment of this application. Figure 2L A logical diagram illustrating the adjustment of the contour shape by overlaying blinking keyframes in an embodiment of this application; Figure 2MThis is a schematic diagram of the sweeping eye-closing process provided in an embodiment of this application; Figure 2N A schematic diagram illustrating the superimposed micro-jitter behavior provided for embodiments of this application; Figure 3 A schematic diagram of the structure of an eye expression image generation device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of this application; Figure 5 This is a schematic diagram of the hardware structure of another computer device that applies an embodiment of this application. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0034] The design concept of the embodiments of this application is briefly introduced below: Artificial intelligence terminals with displays (such as home companion robots, in-vehicle assistants, and public tour guide robots) typically convey emotions and attention to users through digital eyes.

[0035] Previous methods used an offline frame sequence-based approach, which pre-rendered several sets of PNG or GIF frames and played them sequentially after an event was triggered. While simple to implement, this approach suffers from low flexibility, high storage costs, and the need to create a separate complete frame sequence for each new emoji. It also lacks the ability to link with real-time emotional states, making maintenance and expansion difficult.

[0036] Therefore, the current mainstream approach is based on 2D skeletal skinning, which models eye components (such as the eye socket and pupil) in layers and uses bones to drive their scaling and rotation to achieve facial expression changes. However, this approach has limited deformation capabilities, making it difficult to reproduce realistic eyelid curvature and pupil anomalies. Furthermore, the separation of facial expression shape control from dynamic behavior control can easily result in a mask-like or unnatural appearance.

[0037] In view of this, a new method for generating eye expressions is needed to overcome the aforementioned shortcomings. This method includes: first, acquiring the target object's prior state set from the previous frame and the real-time driving set for the current frame; the prior state set refers to the historical expression information of the target object's eye expressions in the previous frame, and the real-time driving set refers to the real-time expression information used to drive the generation of the target object's eye expressions in the current frame; then, based on the eye gaze position obtained from the target object's prior state set from the previous frame and the real-time driving set for the current frame, adjusting the contour shape of the target object's initial eye expression in the current frame; and then, based on dynamic behavior data and a preset data priority, adjusting the obtained candidate eye expressions to obtain the target object's target eye expression in the current frame.

[0038] By adjusting the basic contour shape and overlaying dynamic behavior data, the shape misalignment or temporal disconnection is fundamentally eliminated, enabling the real-time generation of eye expression images with a high degree of coordination between the basic shape and dynamic behavior, thereby improving the delicacy, naturalness, and realism of the target object's eye expression.

[0039] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0040] The eye expression generation method provided in this application can be widely applied in scenarios where emotional interaction needs to be conveyed through dynamic eye movements, such as AI terminals like home companion robots, in-vehicle assistants, and public tour guide robots, to generate eye expression images that match emotional states. In these scenarios, the eye shape of the AI ​​terminal can be dynamically adjusted based on the target object's gaze position shift and dynamic behavior data in adjacent frames, producing a natural interaction effect that closely resembles real human eyes, thereby enhancing the AI ​​terminal's emotional expression capabilities and immersive experience.

[0041] exist Figure 1 One application scenario shown includes two terminal devices 110 and a server 130. The terminal devices 110 establish a communication connection with the server 130 through a wired network or a wireless network.

[0042] Among them, terminal devices 110 include, but are not limited to: mobile phones, computers (such as tablets, laptops, desktop computers, etc.), smart home appliances, smart voice interaction devices (such as smartwatches, smart speakers, etc.), vehicle terminals, aircraft, family companion robots, vehicle assistants, public tour guide robots, etc.

[0043] Server 130 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. This application does not impose any restrictions on these services.

[0044] The interactive display interface 120 of the terminal device 110 displays digital eyes to the target object (such as a family companion robot, a car assistant, a public tour guide robot, etc.) and collects multi-dimensional environmental information around the terminal device 110 in real time (such as the user's voice tone, facial expression, location distance, semantic emotion, etc.). The backend server 130 calls the eye expression generation system to process the multi-dimensional environmental information in real time and generate parameter groups (including emotion parameters and attention parameters) for each frame of the terminal device 110.

[0045] In this process, upon acquiring the parameter set for each frame, the eye expression generation system generates the target object's eye gaze position, target emotion type, and target emotion intensity for that frame. Furthermore, the system also acquires externally inputted emotion control commands (such as emotion locking, transition time settings, etc.), material control commands, and other expression flag information that may affect the eye expression generation results (such as forced blinking, entering sleep state, etc.), thereby forming a real-time driver set for the target object in the current frame.

[0046] The eye expression generation system is based on the real-time driving set of the target object in the current frame, combined with the previous state set of the target object in the previous frame (such as the target emotion type, target emotion intensity, eye gaze position, material playback progress, blink cooldown timer, etc. in the previous frame), adjusts the outline shape of the target object's initial eye expression in the current frame, and superimposes dynamic behavior data to obtain the target eye expression of the target object in the current frame.

[0047] Finally, the eye expression generation system renders the target's eye expression, generating an image of the target object's eye expression in the current frame, and displays the corresponding image on the front-end interactive display interface 120. By integrating multimodal perception output with a unified parameterized interface, emotional state and attentional intent are mapped to the joint control of eye contours, which can simultaneously convey emotional atmosphere and gaze direction, enhancing the interactive intent communication capability and anthropomorphic naturalness of the smart terminal.

[0048] The eye expression generation method proposed in this application is deeply integrated into an eye expression generation system. This system has a flexible deployment architecture, which can be deployed directly on server 130 or on other servers that have established a stable communication connection with server 130, thereby adapting to different hardware environments and network architecture requirements and realizing efficient utilization and elastic expansion of computing resources.

[0049] Combination Figures 2A-2B The illustrated diagram illustrates the process of generating eye expressions of a target object in real time using the method provided in this application.

[0050] S201: Obtain the preceding state set of the target object in the previous frame and the real-time driving set of the current frame; the preceding state set refers to the historical expression information of the target object's eye expression in the previous frame, and the real-time driving set refers to the real-time expression information used to drive the generation of the target object's eye expression in the current frame.

[0051] The standard eye contour refers to the eye contour of the target object under a preset baseline expression, without emotional or motion distortion, and is used as a baseline template for generating various eye expressions. Before executing step 201 and obtaining the initial state of the target object in the current frame, it is first combined with... Figure 2C This paper describes the process of constructing a standard eye contour for a target object. The system uses a morphological modeling layer to uniformly model the geometric shape and spatial position of three core elements: the eye socket contour, the pupil contour, and the pupil highlight, resulting in... Figure 2D The standard eye contour shown. Among them, Figure 2D The focal points shown are for modeling purposes only and will not be directly displayed in actual applications.

[0052] S2011: Construct multiple sets of contour control point sequences, one set of contour control point sequences is used to depict the contour of an eye component.

[0053] First, define multiple sets of contour control point sequences. Specifically, this includes a sequence of contour control points representing the contour of the left eye socket. , representing the sequence of contour control points of the right eye socket. , representing the sequence of contour control points for the left pupil. , representing the sequence of contour control points of the right pupil outline Each set of contour control point sequences It consists of multiple contour control points, arranged in a two-dimensional point array. Represents the i-th contour control point The location coordinates.

[0054] S2012: For each set of contour control point sequences, perform the layer-by-layer interpolation step iteratively until the current layer contour control point sequence contains only two contour control points: perform interpolation processing based on the two adjacent contour control points and the corresponding chord length accumulation sequence of the current layer contour control point sequence to obtain the interpolated control points located between the two adjacent contour control points, and use each interpolated control point generated in the current layer as the contour control point sequence of the next layer; the chord length accumulation sequence is used to represent the approximate cumulative arc length from each contour control point to the first contour control point in the target control point index table.

[0055] Next, the morphological modeling layer uses a contour control point-driven chord length parameterization method and a De Casteljau-style layer-by-layer interpolation method to construct Catmull-Rom spline curves to represent the orbital contour and pupil contour, respectively.

[0056] For each set of contour control point sequences, the layer-by-layer interpolation step is executed iteratively until the current layer contour control point sequence contains only two contour control points: interpolation processing is performed based on the two adjacent contour control points of the current layer contour control point sequence and the corresponding chord length accumulation sequence to obtain the interpolated control points located between the two adjacent contour control points, and each interpolated control point generated in the current layer is used as the contour control point sequence of the next layer.

[0057] The following steps are performed to generate the chord length accumulation sequence of the i-th contour control point: A target control point index table for a single-layer contour control point sequence is constructed using a modular loop approach.

[0058] The eye socket and pupil contours are closed outlines without true start and end points; each contour control point has adjacent contour control points on either side. However, in actual storage, only linear arrays or linear lists can be used to store the contour control points. This linear storage is characterized by having endpoints, with the smallest index value being 0 (corresponding to the start point). The maximum index value is N-1 (corresponding to the termination point). This can lead to index out-of-bounds issues when interpolating the eye outline and pupil outline.

[0059] For example, when calculating the leftmost contour control point When doing so, it is necessary to take its left adjacent point. However, the control point index table using linear storage does not contain -1, and the current index value is less than the minimum index value of 0, resulting in an underflow. For example, when calculating the rightmost contour control point... When doing so, it is necessary to take the point adjacent to its right. However, the maximum index value of the control index table is N-1. Without N, if the current index value is greater than the maximum index value N-1, an overflow occurs.

[0060] To solve this problem, as shown in the formula As shown, this application uses a modular loop method to expand the index values ​​of adjacent points between the start and end points, and constructs a target control point index table for a single contour control point sequence to prevent curve breaks during interpolation and ensure natural connection between the beginning and end of the curve.

[0061] For each contour control point in the original control point index table, perform the following operations: First, take the index value of the i-th contour control point in the original control point index table and perform a modulo operation with the index value of the last contour control point in the table to obtain a first modulo result; then, perform a modulo operation again with the index value of the last contour control point in the table to determine the target contour control point that has a mapping relationship with the i-th contour control point, and use the index value of the target contour control point in the original control point index table as the new index value of the i-th contour control point; finally, construct the target control point index table based on the new index values ​​of all contour control points.

[0062] For example, if the total number of contour control points N is 4, then the starting point... left adjacent point End point The right adjacent point .

[0063] When constructing a continuous curve based on contour control points, a corresponding chord length accumulation sequence needs to be assigned to each control point. If sequence values ​​are assigned arbitrarily, and the density of the contour control point sequence itself is uneven, the constructed curve will exhibit local stretching or compression. For example, if two contour control points are far apart, but their corresponding chord length accumulation sequences have a small interval, the curve will be squeezed between these two points; conversely, for contour control points that are close together, the chord length accumulation sequence intervals will be large, and the curve will be loose.

[0064] This application adopts Formula 1, based on the geometric distance between the i-th contour control point and the (i-1)-th contour control point in the target control point index table. And the cumulative chord length sequence of the first i-1 contour control points Generate the chord length accumulation sequence of the i-th contour control point. where i is a positive integer. It is the string length adjustment factor.

[0065] Formula 1; This application uses a weighted chord length method to calculate the cumulative chord length sequence, which links the interval of the cumulative chord length sequence to the geometric distance between contour control points. The farther apart the contour control points are, the larger the corresponding sequence increment, so that the curves subsequently constructed based on these contour control points better fit the geometric features of the point sequence.

[0066] S2013: Based on each set of contour control point sequences and each set of corresponding multi-layer interpolation control points, a smooth curve contour of each eye component is fitted to form the standard eye contour of the target object in the first frame.

[0067] Taking the modeling of the right eye socket as an example, three contour control points are set to outline the general shape of the eye socket. Interpolation is performed layer by layer on each contour control point. Based on the three contour control points and the interpolated control points generated at each layer, the eye socket is constructed. Figure 2E The complete and smooth contour of the eye socket is shown.

[0068] In the first layer, three interpolation control points are generated, namely: , , , .

[0069] Two interpolation control points are generated in the second layer, namely... , .

[0070] Only one interpolation control point is generated in the third layer. .

[0071] This application employs a layer-by-layer interpolation method to generate smooth, closed boundary curves. It supports both fine-grained local adjustments through control point offsets and overall scaling control through central deformation. Compared to common cubic polynomial and Bezier interpolation, this method offers stronger local controllability. Adjusting a single contour control point only affects the local shape of the curve without impacting the overall contour. This means that under continuous control such as emotion switching and gaze displacement, the eye contour will not exhibit unreasonable protrusions, depressions, or local distortions, thereby improving the stability of the overall geometric shape and the realism of the visual effect. Therefore, it is more suitable for the high-degree-of-freedom, high-frequency change requirements in facial expression modeling.

[0072] This feature enables it to achieve high-precision local deformation and global coordinated adjustment, thereby replacing the traditional discrete frame switching method. It improves the expression resolution from a few preset discrete categories to a continuous range of more than 256 levels, making emotional expression more delicate and gentle (such as the gradual transition from a slight smile to a hearty laugh), and enhancing the user's sense of realism and trust in the target object.

[0073] After constructing the eye socket contour and pupil contour of the target object, Gaussian distribution operation is performed on each pupil contour control point in the pupil contour to generate multiple highlight control points. Based on the multiple highlight control points, the pupil highlight of the target object is constructed.

[0074] The two-dimensional Gaussian model provides continuously adjustable reflection effects. Leveraging the characteristics of the two-dimensional Gaussian distribution function, it can accurately simulate the morphological appearance of pupil highlights under different emotions or lighting conditions, such as the transition levels of brightness and the changes in size, thus completing the modeling of pupil highlights. This modeling method based on the two-dimensional Gaussian distribution function combines morphological controllability with natural rendering, making the presentation of pupil highlights more realistic and natural, conforming to the visual perception of highlights by the human eye, and suitable for dynamic expression synthesis and realism enhancement rendering scenarios.

[0075] After constructing the pupil highlight, you can also dynamically fine-tune the highlight shape by adjusting relevant parameters such as the brightness peak, spatial expansion, and spot center position of the highlight control point. This allows for precise adaptation to the visual characteristics of the eyes in different emotional states (such as brighter and more concentrated highlights when excited, and dimmer and more scattered highlights when tired), or to simulate the reflection differences caused by changes in lighting conditions, thereby further enhancing the realism of eye expressions and scene adaptability.

[0076] The preceding state set is used to record the historical expression information of the target object's eye expression in the previous frame, providing a historical basis for generating the expression in the current frame. This state set includes, but is not limited to, key state information such as the target object's emotion type, emotion intensity, eye gaze position, playback progress, and blink cooldown timer in the previous frame.

[0077] A real-time driver set refers to the real-time facial expression information used to drive the generation of the target object's eye expression in the current frame, providing real-time input for generating the expression in the current frame. This driver set includes, but is not limited to: the target object's eye gaze position in the current frame, the target emotion type and intensity, externally input emotion control instructions (such as emotion locking, transition time settings, etc.), material control instructions, and other expression flag information that may affect the eye expression generation result (such as forced blinking, entering a sleep state, etc.).

[0078] Specifically, the eye expression generation system extracts multi-dimensional environmental information around the target object (such as the user's voice tone, facial expressions, distance between the target object and the user, semantic emotion, etc.), and performs information analysis on the multi-dimensional environmental information to regenerate the attention parameters and emotion parameters for the current frame. The attention parameters are used to reflect behaviors such as gaze focus, attention shift, and dynamic displacement, while the emotion parameters are used to convey facial expression tension and geometric features. Together, they construct a complete expression of the eye contour.

[0079] Subsequently, the system uses a non-linear mapping function to map the attention parameters of the current frame to the eye gaze position of the target object in the current frame, providing a basis for dynamic control of gaze behavior and driving the overall eye to present a dynamic sense of gazing at a certain point or following movement. At the same time, the system also uses a non-linear mapping function to map the emotion parameters of the current frame to the target emotion type (such as gentle, tense, alert, tired, etc.) and target emotion intensity (such as emotion intensity divided into 0-5 levels) of the target object in the current frame, providing a clear target state for emotion switching and expression fusion.

[0080] In addition, the system will also acquire external input emotion control instructions (such as emotion lock, transition time setting, etc.), material control instructions, and other expression marker information that may affect the eye expression generation results (such as forced blinking, entering sleep state, etc.) to help the system more accurately judge the current emotional state and behavioral needs, thereby adjusting the eye contour to generate eye expressions that are closer to real emotional reactions.

[0081] The preceding state set and the real-time driving set together constitute the initial state of the target object in the current frame. If the current frame is the first frame, only the real-time driving set of the target object can be obtained. The real-time driving set is used as the initial state of the target object in the first frame. Based on the real-time driving set, the standard eye contour of the target object in the first frame is adjusted in terms of contour shape and eye expression to generate the target eye expression of the target object in the first frame. This target eye expression is used as the initial eye expression of the target object in the next frame. The relevant expression information of the target object in the first frame is used as the preceding state set of the target object in the next frame.

[0082] To avoid interference between behaviors such as eye contact, emotion switching, material playback, blinking, and significant movement of the object of attention, this application's embodiment inputs the initial state of the current frame into the global state machine. This helps the state machine accurately determine the current behavior pattern and expression requirements, enabling unified scheduling of each sub-module. All sub-module state changes are updated and cleared at the frame level, generating the target eye expression for the target object in the current frame. This ensures the system maintains a stable and predictable operating state over the long term. Compared to multi-module stacking solutions lacking unified management, this application achieves unified scheduling through a global state machine, significantly reducing system complexity and effectively improving maintainability and scalability when adding new behaviors and expressions.

[0083] If the real-time drive fails to extract the target object's eye gaze position in the current frame, this is considered an eye defocusing behavior. The following defocusing search steps must be executed repeatedly until the target object's eye gaze position in the current frame is found or the set search duration is reached: Based on the generated multiple virtual attention parameters, the target object is controlled to move within a preset range of multiple pupil contour control points of the initial eye expression in the current frame; During the movement, multi-dimensional environmental information around the target object is extracted, and based on the extracted multi-dimensional environmental information, the actual attention parameters for the current frame are regenerated. Based on the actual attention parameters of the current frame, the eye gaze position of the target object in the current frame is obtained.

[0084] During the out-of-focus search, the following pupil movement steps are executed repeatedly until the eye fixation position of the target object in the current frame is obtained or the set search duration is reached: Based on the target object's eye gaze position in the previous frame Starting from the point of loss, follow the direction of the last eye movement before the loss. Control the contour control points of the initial eye expression to slide inertia. . It is the inertial strength coefficient, used to control the amplitude of inertial displacement per unit time.

[0085] Subsequently, control each contour control point to stop moving and set the pause time. Based on the inertial reset adjustment of each contour control point, the pupil center point of the target object in the current frame is driven back to the center position of the eye.

[0086] Specifically, based on the search duration since the current frame triggered the search action. Adjust the position of each contour control point. Weight .

[0087] After the eyes lose focus, the system introduces human inertial behavior and a sense of searching through a three-stage process of inertial gliding, brief pause, and blinking back to the center, making the target's gaze appear more like an intentional exploration rather than a mechanical return to the center.

[0088] S202: Based on the eye gaze position obtained from the preceding state set and the real-time driving set, the outline shape of the target object's initial eye expression in the current frame is adjusted to obtain the candidate eye expression of the target object in the current frame.

[0089] The system extracts the eye gaze position of the target object in the previous frame from the preceding state set, and extracts the eye gaze position of the target object in the current frame from the real-time drive set. Based on the extracted eye gaze positions from the two frames, the system obtains the gaze position offset of the target object in the two adjacent frames.

[0090] The gaze position refers to the position coordinates of the object of interest in the target object within the screen coordinate system. Based on the gaze position offset of the target object in two adjacent frames, it is determined whether the object of interest of the target object has changed drastically between adjacent frames.

[0091] When the gaze position of the target object in two adjacent frames deviates from the position offset threshold, the first movement mode is adopted to adjust the outline shape of the target object's initial eye expression in the current frame and obtain the candidate eye expression of the target object in the current frame.

[0092] When the gaze position of the target object shifts more than the position shift threshold between two adjacent frames, it indicates that the position of the object of attention has changed significantly. In this case, a time-sharing movement mode should be adopted to simulate rapid eye saccades and adjust the outline of the target object's initial eye expression in the current frame to obtain the candidate eye expressions of the target object in the current frame.

[0093] Specifically, based on the duration of the gaze jump from the current frame to the trigger time and the gaze position of the target object in the current frame, the positions of the pupil contour control points of the target object's initial eye expression in the current frame are adjusted. After confirming that the pupil contour control points have been adjusted, based on the duration of the jump and the gaze position of the target object in the current frame, the positions of the orbital contour control points of the target object in the current frame are further adjusted to form... Figure 2F The natural visual effect shown is that the pupil moves first and the eye socket follows, avoiding the mask-like feeling caused by rigid overall translation.

[0094] During the position adjustment phase, the system does not directly modify the position coordinates of the contour control points. Instead, it first uses Formula 2 to apply affine displacement to the overall orbital contour and pupil contour, respectively, to obtain the orbital displacement for each eye (the left and right eyes are calculated independently). Pupil displacement .

[0095] Formula 2; In Formula 2, g represents the eye gaze position of the target object in the current frame. This represents the sensitivity coefficient of horizontal displacement of the orbital contour. This represents the sensitivity coefficient of the eye socket contour in the vertical direction. This is the basic offset for the orbital contour, used to compensate for individual differences or initial posture; This represents the sensitivity coefficient of pupil contour displacement in the horizontal direction. This represents the sensitivity coefficient of pupil contour displacement in the vertical direction. This is the basic bias term for the pupil outline.

[0096] To conform to the characteristics of human gaze behavior, the following constraints must be satisfied: This ensures that, under the same gaze position, the movement of the pupil outline is greater than the movement of the eye socket outline, creating a natural visual effect where the pupil moves first and the eye socket follows, avoiding the mask-like feeling caused by rigid overall translation.

[0097] The aforementioned displacement is not applied directly to the contour control points. Instead, Formula 3 is used to apply an affine transformation to the entire sequence, using the contour center point c of the contour control point sequence as a reference, to obtain the target position of each contour control point in the current frame. Among them, in formula 3 It is a scaling matrix related to the gaze position, used to simulate effects such as near and far focusing.

[0098] Formula 3; Finally, using Formula 4, the duration of the jump is... Substitute the buffer function to obtain the time-varying result. Normalized weights of change Based on the ease-out time and the target position of each contour control point in the current frame, the position of each contour control point is adjusted to form a gaze shift behavior similar to that of humans.

[0099] Formula 4; When the gaze position of the target object in two adjacent frames is not greater than the position offset threshold, it means that the position of the object of attention has not changed significantly. The second movement mode (i.e., synchronous movement mode) should be adopted to adjust the outline shape of the initial eye expression and obtain the candidate eye expression of the target object in the current frame.

[0100] Specifically, based on the duration of the gaze jump from the current frame to the trigger time and the gaze position of the target object in the current frame, the positions of each pupil contour control point and each orbital contour control point of the target object's initial eye expression in the current frame are adjusted to form... Figure 2G The natural visual effect shown is achieved by moving the pupil and eye socket simultaneously.

[0101] S203: Based on dynamic behavior data and preset data priority, adjust the eye expressions of candidate eye expressions to obtain the target eye expression of the target object in the current frame; dynamic behavior data refers to the data that triggers changes in the eye expression of the target object.

[0102] To avoid interference between behaviors such as eye contact, emotion switching, media playback, blinking, and significant movement of the object of attention, this embodiment uses a global state machine to uniformly schedule each submodule. By default, media control commands have higher priority than emotion switching; when receiving dynamic behavior data of the same type sequentially, the data received later has higher priority than the data received earlier. Furthermore, the system also supports customizing the priority order of dynamic behavior data according to actual needs.

[0103] First, based on the facial expression material information obtained from the real-time driving center, the candidate eye expressions are fused to obtain the eye expressions of the target object in the current frame.

[0104] The emoticon material information includes instructions for playing the material, exiting the material, switching the material, and pausing the material, and explains the specific execution process of each instruction.

[0105] If the facial expression material information in the real-time drive set is a material playback instruction, the progress of the material playback progress of the previous frame in the preceding state set is updated. When the material playback progress of the current frame indicates that the current frame is in the material playback stage, the facial expression material of the first frame and the candidate eye expressions are processed by facial expression fusion to obtain the eye expression of the target object in the current frame.

[0106] Based on the duration of playback since the current frame was triggered ( ) ) and total playback time Get the playback progress of the material .

[0107] The clamp function is used to clamp an input value ( The input value is constrained to the range between the minimum value (0) and the maximum value (1) to prevent the value from exceeding this range. The result is returned as a value within the range [0, 1]. If the input value is less than the minimum value, the minimum value is returned; if the input value is greater than the maximum value, the maximum value is returned; if the input value is within the range [0, 1], the input value itself is returned directly (no constraint required, preserving the original value).

[0108] If the first frame of the expression material is not in a closed-eye state, then based on the playback progress of the material in the current frame, a first transition weight is obtained to control the transition speed curve. Based on the first transition weight, the first frame of the expression material and the candidate eye expressions are merged to generate the target object's eye expression in the current frame.

[0109] Based on the playback progress of the source material, a first transition weight is obtained to control the transition speed curve. . Specify boundary conditions (Corresponding to the start of playback) (Corresponding to the end of playback). To prevent weights from exceeding a reasonable range, avoid anomalies such as transition speed being too fast, transition speed being too slow, or transition effect overflow (e.g., excessive contour deformation), and ensure the consistency and stability of transition effects under different materials and different playback scenarios.

[0110] If the first frame of the facial expression is not showing the eyes closed, then use... , the first frame of the emoji material and candidate eye expressions Perform contour blending to obtain the eye expression of the target object in the current frame. .

[0111] For example, the candidate eye expression displays a crescent shape characteristic of a smile, while the first frame's expression material is heart-shaped. By merging the two, the result is as follows: Figure 2H The image shows a blended eye contour. The heart-shaped element is used to express the target audience's excitement and exhilaration.

[0112] If the first frame of the expression material shows the eyes closed, then the candidate eye expression is replaced with the first frame of the expression material to obtain the target object's eye expression in the current frame.

[0113] If the first frame of the facial expression is in a closed-eye state, then use... Then, the candidate eye expression is replaced with the expression material of the first frame, and the eye expression of the target object in the current frame is obtained.

[0114] If the facial expression material information in the real-time drive set is a material exit instruction, the material exit progress of the previous frame in the preceding state set is updated. When the material exit progress of the current frame indicates that the current frame is in the material exit stage, the facial expression material of the last frame and the candidate eye expression are processed by facial expression fusion to obtain the eye expression of the target object in the current frame.

[0115] An exit command will be generated upon completion of playback of the clip or in response to a user's trigger action on the clip exit control. This is based on the elapsed duration since the clip exit was triggered in the current frame (…). ) and total exit time Get material exit progress .

[0116] Based on the exit progress of the material in the current frame, a second transition weight is obtained to control the transition speed curve. . Specify boundary conditions (Corresponding to the start of playback) (Corresponding to the end of playback). To prevent weights from exceeding a reasonable range, avoid anomalies such as transition speed being too fast, transition speed being too slow, or transition effect overflow (e.g., excessive contour deformation), and ensure the consistency and stability of transition effects under different materials and different playback scenarios.

[0117] Based on the second transition weight, the eye expression from the last frame is fused. and candidate eye expressions Generate the eye expression of the target object in the current frame. .

[0118] For example, the candidate eye expression is presented as a heart shape, while the final frame's expression material is crescent-shaped. By merging the two, the result is as follows: Figure 2I The image shown depicts the expression in the eyes. This crescent-shaped element is used to express the target audience's shift from initial excitement and exhilaration to a calmer, happier mood after the video ends.

[0119] If the real-time driver sets the facial expression material information as a material switching instruction, it obtains new eye expression material, and performs expression fusion processing on the new first frame eye expression material and candidate eye expressions to obtain the target object's eye expression material in the current frame.

[0120] For example, such as Figure 2J As shown, the system receives a material switching instruction, obtains new eye expression material, and performs expression fusion processing on the new first frame eye expression material and candidate eye expressions, changing the original heart pattern to a frustrated expression, thus achieving rapid emotion switching for the target object.

[0121] If the real-time driver receives a pause command for the facial expression material, the fusion step is paused, and the outline of the candidate eye expression remains unchanged until a new playback command, exit command, or switching command is received. Only then will the facial expression fusion step be executed again to generate the fused eye expression material.

[0122] Through the aforementioned differentiated interpolation control mechanism based on material control instructions, this application achieves dynamic behavior adjustment of different eye expression materials during playback and exit without disrupting the unified interpolation framework. It is controllable in the time dimension, spatial dimension, and dynamic response dimension, effectively eliminating visual jumps and abrupt switching, and significantly improving the naturalness and coherence of material switching.

[0123] In addition to using the linear function mentioned above to calculate the first transition weight With the second transition weight In addition, first-order smoothing functions, third-order smoothing functions, fifth-order smoothing functions, or lookup table functions can be used to calculate transition weights to achieve different transition effects. The first transition weight is used as an example. For example, the formula for calculating the transition weight using a first-order smoothing function is as follows: The transition weight calculation formula using the cubic smoothing function is as follows: The transition weight calculation formula using the fifth-order smoothing function is as follows: The transition weight calculation formula using the lookup table function is as follows: .

[0124] Furthermore, to control the transition speed curve of the profile, group interpolation or channel interpolation can be performed on the profile. Assume a profile curve consists of multiple subsets. It can be a different subset. Configure different transition weights Generate the eye expression of the target object in the current frame. This creates a transition path between different areas such as the upper eye socket, lower eye socket, and corner of the eye, enhancing visual naturalness.

[0125] First Transition Weight With the second transition weight Not only can it control the transition speed curve, but it can also... Controls enable or disable the rebound switch to achieve a gradual rebound transition effect.

[0126] In the weight out-of-bounds bounce mode, the transition weight of the j-th subset use or Its value range has been expanded to , To allow weights to exceed the normal [0, 1] range, the calculation formula for fusing emoji materials is as follows: .

[0127] Adding a damped oscillation term to simulate a natural rebound effect, the calculation formula for integrating facial expression materials then becomes... .in, The rebound amplitude, The attenuation coefficient is... D represents the frequency. D represents the rebound direction, which is the default. And predefine the deformation direction. When When the bounce value is zero, the bounce switch is disabled, and the gradual bounce transition effect is turned off.

[0128] Secondly, based on the emotional state information obtained from the preceding state set and the real-time driving set, the eye expressions of the material are processed by emotion fusion to obtain the emotional eye expressions of the target object in the current frame.

[0129] Based on the emotion type and intensity obtained from the preceding state set and the real-time driving set, the emotion switching result of the target object in two adjacent frames is obtained.

[0130] When the emotion switching result of the target object in two adjacent frames matches any one of the emotion switching rules, it indicates that the target object's emotion has changed drastically. Without superimposing the blink keyframe, the outline shape of the target object's eye expression is adjusted based solely on the target emotion type and intensity of the target object in the current frame to obtain the target object's emotional eye expression in the current frame, in order to simulate real human emotional fluctuations.

[0131] The emotion switching rules include: the target being switched to a happy emotion, and the intensity of the happy emotion is at level 4 or 5 (the higher the intensity level, the stronger the emotional expression); the target being switched to a sad emotion, and the intensity of the happy emotion is at level 4 or 5; the target being switched from any emotion type to a surprised emotion.

[0132] To achieve eye geometry features for different emotion types, this application establishes a correspondence between each emotion type and its corresponding emotional intensity and a prior model. It pre-stores multiple continuously varying sequences of contour control points, each containing multiple control points. This allows for updates to the coordinates, curvature distribution, and overall geometric shape of the control points, thereby driving dynamic changes in the appearance. Specific deformations may include, but are not limited to, adjustments to visual elements such as eyelid opening and closing amplitude, pupil shape variation, and highlight blur level, to achieve a more emotionally expressive appearance response.

[0133] Taking happiness as an example, the system meticulously divides it into different emotional intensities, such as 30 or 50, with each intensity exhibiting significant differences in its expressive form. From the slight upturn of the corners of the mouth to the degree of relaxation in the corners of the eyes, and even the dilation and contraction of the pupils, each emotion displays a hierarchical change as the intensity increases. This grading method breaks away from the "jumps" in emotional expression found in traditional approaches, allowing for a natural transition in form when emotions shift from low to high intensity, laying the foundation for subsequent subtle "emotional gradations."

[0134] Next, combined Figure 2K The process of adjusting the outline shape of the eye expression of the target object based on the target emotion type and intensity in the current frame is as follows.

[0135] First, obtain multiple sets of contour control point sequences corresponding to the target emotion type e and target emotion intensity l of the target object in the current frame. .

[0136] Secondly, for two adjacent sets of contour control point sequences, the following operations are performed respectively: interpolation processing is performed on the two adjacent sets of contour control point sequences to obtain the interpolated control point sequence under the emotional transition state.

[0137] Formula 5 illustrates the specific implementation of interpolation processing for a set of contour control point sequences, which calculates the duration of the current frame since the trigger time of the current emotion switch. Substitute the buffer function to obtain the time-varying result. Normalized weights of change It is used to enhance the natural movement effect of "gradual changes in the initial stage and rapid changes in the later stage".

[0138] Formula 5; Finally, based on multiple sets of contour control point sequences and multiple sets of interpolation control point sequences, the contour shape of the eye expression in the material is adjusted continuously multiple times. By continuously updating the position coordinates of the contour control points, the eye contour gradually approaches the target shape, thereby achieving a smooth and natural eye deformation effect. The eye expression obtained from the last adjustment is used as the emotional eye expression of the target object in the current frame.

[0139] When the emotion switching result indicates that the target object's emotion has not changed drastically, a blinking keyframe is superimposed to adjust the outline shape of the eye expression driven by the blinking behavior, thereby obtaining the target object's emotional eye expression in the current frame and masking the abruptness of the outline switching.

[0140] Combination Figure 2L When the current frame is determined to be in the blinking phase, the blinking keyframes and adjacent keyframes associated with the blinking process in the current frame are obtained. Interpolation processing is performed on the blinking contour control points contained in the associated blinking keyframes and adjacent keyframes to obtain the interpolated control point sequence in the blinking transition state. Finally, based on the interpolated control point sequence in the blinking transition state, the contour shape of the eye expression in the footage is adjusted. By continuously updating the position coordinates of the contour control points, the eye contour gradually approaches the target shape, thus obtaining the emotional eye expression of the target object in the current frame.

[0141] Blinking behavior mainly includes eye closing and eye opening; therefore, the blinking process can be discretized into multiple blinking keyframes, using a formula... This will count the duration of blinks since the current blink was triggered in the current frame. Map to consecutive index values ​​u. Then use the formula. By integer index value With weight Interpolation processing is performed on the associated blink keyframes and adjacent keyframes to obtain the interpolation control point sequence in the blink transition state, so as to achieve smooth and controllable blink closing and eye opening animation effects.

[0142] For example, total blinking time It lasts 0.6 seconds and has 4 blink keyframes. These correspond to the initial open-eye state ( ), half-closed eyes state, fully closed eyes state ( (and the state of reopening eyes.)

[0143] Assuming the duration of blinks since the current blink was triggered in the current frame. The time is 0.2 seconds, which is during the blinking process. Calculations show that the continuous index value... Integer index value (Corresponding to the first blink keyframe) and weights .

[0144] Interpolation is performed on the first blink keyframe and the adjacent second blink keyframe associated with the blink process to obtain the interpolated control point sequence of the current frame in the blink transition state. Assuming the position coordinates of the i-th blink contour control point in the first blink keyframe are (5,10), and the position coordinates of the same control point in the second blink keyframe are (5,15), then the position coordinates of this point 0.2 seconds after the blink is completed are... .

[0145] Extract the remaining blink cooldown time from the blink cooldown timer in the previous state set. If the blinking interval between two adjacent frames is less than the blink cooldown time, the blink cooldown mechanism is activated, preventing the superimposed blinking behavior from occurring in the current frame. If the blinking interval between two adjacent frames is not less than the blink cooldown time, the blink cooldown mechanism is deactivated, allowing the superimposed blinking behavior to occur in the current frame, and the blink cooldown timer is restarted to begin a new round of cooldown timing.

[0146] This application employs a blink cooling mechanism to prevent blink events from being triggered frequently within a short period of time. For example, if the interval between blink events in two adjacent frames is less than the blink cooling time, the blink cooling mechanism is triggered, prohibiting the blink event from being triggered again, until the interval between blink events is no less than the blink cooling time, at which point the cooling is lifted, allowing the blink event to be superimposed again.

[0147] Furthermore, this application sets whether a blinking behavior triggers a blink cooling-off mechanism and whether it is affected by the blink cooling-off mechanism for different types of blinking behavior, thereby achieving priority differentiation of blinking behavior. For example, the "normal blink" type will trigger the cooling-off mechanism, while the "emotional blink" type (such as a happy blink) will not trigger the cooling-off mechanism, to ensure that emotion-related blinking behaviors can be responded to first. If the blinking behavior has ended, the blinking state and blink cooling-off timer are reset to prepare for the next blinking behavior.

[0148] By combining time mapping, keyframe interpolation, and cooldown control, a smooth animation effect from opening the eyes to closing the eyes and then opening the eyes is achieved. At the same time, the blink cooldown mechanism distinguishes the trigger priority of different types of blinking behavior, thereby avoiding blinking behavior too frequently.

[0149] When it is determined that the eye gaze position of the target object changes drastically between two adjacent frames, the following operations are performed sequentially on each eye expression of the target object: based on the duration of the jump and the total duration of the jump from the current frame to the trigger time of the current gaze jump, the closure weight of each eye socket contour control point in the eye expression is obtained; based on the closure weight of each contour control point, the position of each eye socket contour control point in the eye expression is adjusted to obtain the emotional eye expression of the target object in the current frame.

[0150] Combination Figure 2M When it is determined that the gaze position of the target object changes drastically between two adjacent frames, the following operations are performed sequentially on the eye expression of each clip of the target object: Based on the duration of the gaze jump since the current frame was triggered. Total duration of transition Get the global blink progress of the current frame. Then, based on the global blink progress, the closure weights of each eye socket contour control point in the material's eye expression are obtained. .in, This indicates the overall blinking progress. Once it increases to a certain extent, the kth orbital contour control point will begin to participate in closure and will linearly increase from 0 to 1 over a period of time.

[0151] Subsequently, based on the closure weights of each contour control point, the positions of the eye socket contour control points in the source material are adjusted to obtain the emotional eye expression of the target object in the current frame. Specifically, the eye socket contour control points located in the upper eye socket are determined using a formula... Update the position coordinates of the contour control points; the contour control points located in the lower eye socket are updated using a formula. Update the position coordinates of the contour control points.

[0152] By progressively advancing the closing displacement of the upper and lower eye sockets along a sequence of contour control points, a smooth blinking effect is achieved instead of a rigid, overall closure. Specifically, when the gaze rapidly shifts from the far left to the far right, a closing displacement is applied segment by segment to the vertical coordinates of the upper and lower eye sockets from right to left. The eyelids create an effect similar to air resistance or eye-following occlusion along the direction of movement, forming a sweeping eye-closing process where the right side closes first, followed by the left. At the end of the large displacement phase, time reversal or symmetrical interpolation is performed, causing the eyelids to reopen sequentially from right to left, strictly aligned with the moment the eyeball movement ends. This design significantly enhances the anthropomorphic performance in scenarios with large gaze shifts, enabling the system to produce an animation effect similar to the natural squinting of the eyes when following an object of attention.

[0153] This application can also adjust the intensity level of micro-trembling behavior parameters (e.g., frequency parameters, amplitude parameters) by combining a preset mapping relationship with the target emotion type and intensity, generate a perturbation vector, and superimpose the perturbation vector onto the pupil contour to form a... Figure 2N The dynamic effects shown simulate the subtle tremors of the pupils when the eyes are still and gazing, enhancing the anthropomorphism of the gaze and increasing the vividness and breathiness of the eyes.

[0154] Finally, based on the facial expression flag information obtained from the real-time driving set, the emotional eye expression is processed by behavior overlay to obtain the target eye expression of the target object in the current frame.

[0155] If the facial expression flag information obtained by the real-time driver is a forced blink or entering a sleep state, the contour shape of the emotional eye expression is adjusted by blink behavior-driven adjustment, simulating the continuous change of the eyelid from slow closing to rapid opening during natural blinking, to obtain the target eye expression of the target object in the current frame, thereby improving the visual realism and expressiveness of the target object's special expression commands and sleep state.

[0156] After obtaining the target eye expression of the target object in the current frame, it is transmitted to the image rendering layer for image rendering to generate the corresponding eye expression image. At the same time, it is used as the initial eye expression for the next frame to ensure the continuity and smooth transition of expression changes.

[0157] During the image rendering stage, the system updates the position coordinates of each contour control point in real time, and then dynamically reconstructs the contour curves and geometric structures based on the updated position coordinates. This process does not rely on any offline frame resources (such as GIF sequences, sprite sheets) or texture switching, thus eliminating the dependence on pre-stored resources in traditional solutions from the underlying mechanism, significantly improving rendering efficiency and the flexibility of dynamic performance.

[0158] Subsequently, the system writes the adjusted control point position coordinates and position vectors into the vertex buffer of the graphics processing unit (GPU) in real time, and the graphics pipeline completes the complete rendering process on the GPU: by reconstructing a smooth closed contour curve, pixel filling is performed after geometric restructuring, and finally the eye expression image of the current frame is generated.

[0159] The morphological calculation and behavior scheduling modules designed in this application both adopt a lightweight structure, supporting control point update calculations of less than 0.1 ms on the CPU side and curve rendering processes of less than 4 ms on the GPU side. This rendering link supports real-time output at a frame rate of 60 (Frames Per Second, FPS), which can not only ensure the smoothness of facial expression dynamics, but also adapt to mid-to-low-end embedded platforms such as Cortex-A55, meeting the operating requirements of lightweight terminal devices.

[0160] Based on the same inventive concept as the above-described method embodiments, this application also provides an eye expression generation device. For example... Figure 3 As shown, the eye expression generation device 300 may include: The state acquisition module 301 is used to acquire the previous state set of the target object in the previous frame and the real-time driving set of the current frame; the previous state set refers to the historical expression information of the target object's eye expression in the previous frame, and the real-time driving set refers to the real-time expression information used to drive the generation of the target object's eye expression in the current frame. The shape adjustment module 302 is used to adjust the outline shape of the initial eye expression of the target object in the current frame based on the eye gaze position obtained by the preceding state set and the real-time drive set, so as to obtain the candidate eye expression of the target object in the current frame. The behavior overlay module 303 is used to adjust the eye expressions of the candidate eye expressions based on dynamic behavior data and preset data priorities, so as to obtain the target eye expression of the target object in the current frame; the dynamic behavior data refers to the data that triggers the change of the target object's eye expression.

[0161] Optionally, the shape adjustment module 302 is used for: Extract the eye gaze position of the target object in the previous frame from the preceding state set, and extract the eye gaze position of the target object in the current frame from the real-time drive set; Based on the extracted eye gaze positions of the two frames, the gaze position offset of the target object in the two adjacent frames is obtained; When the gaze position of the target object in two adjacent frames is offset by more than the position offset threshold, the first movement mode is adopted to adjust the outline shape of the initial eye expression of the target object in the current frame and obtain the candidate eye expression of the target object in the current frame. When the gaze position offset of the target object in two adjacent frames is not greater than the position offset threshold, the second movement mode is adopted to adjust the contour shape of the initial eye expression and obtain the candidate eye expression of the target object in the current frame.

[0162] Optionally, the shape adjustment module 302 is used for: Based on the duration of the gaze jump from the current frame to the trigger time of this gaze jump and the gaze position of the target object in the current frame, the positions of the control points of each pupil contour of the target object's initial eye expression in the current frame are adjusted. After confirming that the control points of each pupil contour have been adjusted, based on the jump duration and the gaze position of the target object in the current frame, the position of each orbital contour control point of the target object in the current frame is further adjusted.

[0163] Optionally, the shape adjustment module 302 is used for: Based on the duration of the gaze jump from the current frame to the trigger time of this gaze jump and the gaze position of the target object in the current frame, the positions of each pupil contour control point and each eye socket contour control point of the target object's initial eye expression in the current frame are adjusted.

[0164] Optionally, the behavior overlay module 303 is used for: Based on the facial expression material information obtained from the real-time driving set, the candidate eye expressions are fused to obtain the eye expressions of the target object in the current frame. Based on the emotional state information obtained from the preceding state set and the real-time driving set, the eye expressions of the material are processed by emotion fusion to obtain the emotional eye expressions of the target object in the current frame. Based on the facial expression flag information obtained from the real-time driving set, the emotional eye expression is subjected to behavior superposition processing to obtain the target eye expression of the target object in the current frame.

[0165] Optionally, the behavior overlay module 303 is used for: If the facial expression material information in the real-time drive set is a material playback instruction, the progress update is performed on the material playback progress of the previous frame in the preceding state set. When the material playback progress of the current frame indicates that the current frame is in the material playback stage, the facial expression material of the first frame and the candidate eye expression are processed by facial expression fusion to obtain the material eye expression of the target object in the current frame. If the facial expression material information in the real-time drive set is a material exit instruction, the material exit progress of the previous frame in the preceding state set is updated. When the material exit progress of the current frame indicates that the current frame is in the material exit stage, the last frame facial expression material and the candidate eye expression are processed by facial expression fusion to obtain the material eye expression of the target object in the current frame. If the expression material information in the real-time drive set is a material switching instruction, a new eye expression material is obtained, and the new first frame eye expression material and the candidate eye expression are processed by expression fusion to obtain the target object's material eye expression in the current frame.

[0166] Optionally, the behavior overlay module 303 is used for: If the first frame of the expression material is not in a closed-eye state, then based on the playback progress of the material in the current frame, a first transition weight is obtained to control the transition speed curve, and based on the first transition weight, the first frame of the expression material and the candidate eye expression are fused to generate the target object's eye expression in the current frame. If the first frame of the expression material shows the eyes closed, then the candidate eye expression is replaced with the first frame of the expression material to obtain the target object's eye expression in the current frame.

[0167] Optionally, the behavior overlay module 303 is used for: Based on the material exit progress of the current frame, a second transition weight is obtained to control the transition speed curve; Based on the second transition weight, the last frame's expression material and the candidate eye expressions are fused to generate the target object's eye expression material in the current frame.

[0168] Optionally, the behavior overlay module 303 is further configured to: If the expression material information in the real-time driver set is a material pause playback command, the fusion step is paused, and the outline shape of the candidate eye expression remains unchanged.

[0169] Optionally, the behavior overlay module 303 is used for: Based on the emotion type and emotion intensity obtained from the preceding state set and the real-time driving set, the emotion switching result of the target object in two adjacent frames is obtained. When the emotion switching result of the target object in two adjacent frames indicates that the target object's emotion has changed drastically, the outline shape of the eye expression of the material is adjusted based on the target emotion type and target emotion intensity of the target object in the current frame to obtain the emotional eye expression of the target object in the current frame. When the emotion switching result indicates that the target object's emotion has not changed drastically, the outline shape adjustment of the eye expression of the material driven by blinking behavior is performed to obtain the emotional eye expression of the target object in the current frame.

[0170] Optionally, the behavior overlay module 303 is used for: Obtain multiple sets of contour control point sequences corresponding to the target emotion type and intensity of the target object in the current frame; For two adjacent sets of contour control point sequences, perform the following operations respectively: perform interpolation processing on the two adjacent sets of contour control point sequences to obtain the interpolated control point sequence under the emotional transition state; Based on multiple sets of contour control point sequences and multiple sets of interpolation control point sequences, the contour shape of the eye expression of the material is adjusted multiple times in succession, and the eye expression obtained by the last adjustment is taken as the emotional eye expression of the target object in the current frame.

[0171] Optionally, the behavior overlay module 303 is used for: When it is determined that the current frame is in the blinking phase, obtain the blinking keyframes and adjacent keyframes associated with the current frame during the blinking process; Interpolation processing is performed on the associated blink keyframes and the blink contour control points contained in adjacent keyframes to obtain the interpolation control point sequence in the blink transition state. Based on the interpolation control point sequence during the blink transition, the outline shape of the eye expression in the material is adjusted to obtain the emotional eye expression of the target object in the current frame.

[0172] Optionally, the behavior overlay module 303 is used for: When it is determined that the eye gaze position of the target object changes drastically between two adjacent frames, perform the following operations on the eye expression of each clip of the target object in sequence: Based on the duration of the jump and the total duration of the jump in the current frame from the trigger time of the current gaze jump, the closure weight of each eye socket contour control point in the eye expression of the material is obtained. Based on the closure weight of each contour control point, the position of each eye socket contour control point in the material's eye expression is adjusted to obtain the emotional eye expression of the target object in the current frame.

[0173] Optionally, if the real-time drive set fails to extract the eye gaze position of the target object in the current frame, the state acquisition module 301 repeatedly executes the following defocus search steps until the eye gaze position of the target object in the current frame is found or the set search duration is reached: Based on the generated multiple virtual attention parameters, the target object is controlled to move within a preset range of multiple pupil contour control points of the initial eye expression in the current frame; During the movement, multi-dimensional environmental information around the target object is extracted, and based on the extracted multi-dimensional environmental information, the actual attention parameters for the current frame are regenerated. Based on the actual attention parameters of the current frame, the eye gaze position of the target object in the current frame is obtained.

[0174] Optionally, during the out-of-focus search process, the state acquisition module 301 cyclically executes the following pupil movement steps until the eye fixation position of the target object in the current frame is obtained or the set search duration is reached: Taking the target object's eye gaze position in the previous frame as the starting point of loss, the contour control points of the initial eye expression are controlled to slide inertially along the last eye movement direction before loss. The system controls each contour control point to stop moving for a set duration, and adjusts the inertial reset of each contour control point to drive the pupil center point of the target object in the current frame back to the center position of the eye.

[0175] Optionally, the behavior overlay module 303 is used for: If the expression flag information obtained by the real-time drive set is a forced blink or entering a sleep state, the contour shape of the emotional eye expression is adjusted by blink behavior to obtain the target eye expression of the target object in the current frame.

[0176] Optionally, the behavior overlay module 303 is further configured to: Extract the remaining blink cool-down time from the blink cool-down timer in the preceding state set; If the blinking behavior trigger interval between two adjacent frames of the target object is less than the blinking cooldown time, the blinking cooldown mechanism is activated to prevent the superposition of blinking behavior in the current frame. If the blinking behavior trigger interval between two adjacent frames of the target object is not less than the blinking cooldown time, the blinking cooldown mechanism is deactivated, allowing the blinking behavior to be superimposed in the current frame, and the blinking cooldown timer is restarted to start a new round of cooldown timing.

[0177] Optionally, the eye expression generation device 300 further includes a contour construction module 304. Before acquiring the previous state set of the target object in the previous frame and the real-time driving set of the current frame, the contour construction module 304 performs the following steps to construct the standard eye contour of the target object in the first frame: Construct multiple sets of contour control point sequences, one set of contour control point sequences is used to depict the contour of an eye component; For each set of contour control point sequences in the standard eye contour, a layer-by-layer interpolation step is performed iteratively until the current layer contour control point sequence contains only two contour control points: interpolation processing is performed based on the two adjacent contour control points of the current layer contour control point sequence and the corresponding chord length accumulation sequence to obtain the interpolated control points located between the two adjacent contour control points, and each interpolated control point generated in the current layer is used as the contour control point sequence of the next layer; the chord length accumulation sequence is used to represent the approximate cumulative arc length from each contour control point to the first contour control point in the target control point index table; Based on each set of contour control point sequences and each set of corresponding multi-layer interpolation control points, a smooth curve contour of each eye component is fitted to form the standard eye contour of the target object in the first frame.

[0178] Optionally, the contour construction module 304 performs the following steps to generate the chord length accumulation sequence of the i-th contour control point: A target control point index table for a single-layer contour control point sequence is constructed using a modular loop approach. Based on the geometric distance between the i-th contour control point and the (i-1)-th contour control point in the target control point index table, and the cumulative chord length sequence of the first (i-1) contour control points, the cumulative chord length sequence of the i-th contour control point is generated, where i is a positive integer.

[0179] The contour construction module 304 is used for: For each contour control point in the original control point index table, perform the following operations: Take the index value of the i-th contour control point in the original control point index table and perform a modulo operation with the index value of the last contour control point in the table to obtain a first modulo result; Perform another modulo operation with the first modulo result and the index value of the last contour control point in the table to determine a target contour control point that has a mapping relationship with the i-th contour control point; Use the index value of the target contour control point in the original control point index table as the new index value of the i-th contour control point. Based on the new index values ​​of all contour control points, construct the target control point index table.

[0180] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware components.

[0181] Having introduced the eye expression generation method and apparatus according to exemplary embodiments of this application, we will now introduce a robot according to another exemplary embodiment of this application.

[0182] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0183] Based on the same inventive concept as the above-described method embodiments, this application also provides a robot. In one embodiment, the computer device may be a server, such as... Figure 1 The server 130 is shown. In this embodiment, the computer device is structured as follows: Figure 4 As shown, it may include at least a memory 401, a communication module 403, and at least one processor 402.

[0184] The memory 401 is used to store computer programs executed by the processor 402. The memory 401 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and programs required to run instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0185] Memory 401 may be volatile memory, such as random-access memory (RAM); memory 401 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 401 may be any other medium capable of carrying or storing a desired computer program having the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 401 may be a combination of the above-described memories.

[0186] The processor 402 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 402 is used to implement the aforementioned eye expression generation method when it calls the computer program stored in the memory 401.

[0187] The communication module 403 is used to communicate with terminal devices and other servers.

[0188] This application embodiment does not limit the specific connection medium between the memory 401, communication module 403, and processor 402 described above. This application embodiment... Figure 4The memory 401 and the processor 402 are connected via a bus 404, and the bus 404 is in Figure 4 The diagram uses thick lines to describe the connections between other components; these are for illustrative purposes only and should not be considered limiting. The 404 bus can be divided into address bus, data bus, control bus, etc. For ease of description, Figure 4 It is described using only a thick line, but does not indicate that there is only one bus or one type of bus.

[0189] The memory 401 stores a computer storage medium, which stores computer-executable instructions for implementing the eye expression generation method of this application embodiment. The processor 402 is used to execute the above-described eye expression generation method, such as... Figure 2A As shown.

[0190] In another embodiment, the robot can also be other types of robots, such as... Figure 1 The terminal device 110 is shown. In this embodiment, the robot's structure can be as follows: Figure 5 As shown, it includes components such as: communication component 510, memory 520, display unit 530, camera 540, sensor 550, audio circuit 560, Bluetooth module 570, and processor 580.

[0191] The communication component 510 is used to communicate with the server. In some embodiments, it may include a Wireless Fidelity (WiFi) module, which is a short-range wireless transmission technology, and the electronic device can send and receive information through the WiFi module.

[0192] The memory 520 can be used to store software programs and data. The processor 580 executes various functions of the terminal device 110 and performs data processing by running the software programs or data stored in the memory 520. The memory 520 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. The memory 520 stores an operating system that enables the terminal device 110 to run. In this application, the memory 520 may store the operating system and various application programs, and may also store a computer program that executes the eye expression generation method of the embodiments of this application.

[0193] The display unit 530 can also be used to display information input by the object or information provided to the object, as well as various menus of the terminal device 110, in a graphical user interface (GUI). Specifically, the display unit 530 may include a display screen 532 disposed on the front of the terminal device 110. The display screen 532 may be configured as a liquid crystal display, a light-emitting diode, or the like. The display unit 530 can be used to display the defect detection interface, model training interface, etc., as described in the embodiments of this application.

[0194] The display unit 530 can also be used to receive input digital or character information and generate signal inputs related to object settings and function control of the terminal device 110. Specifically, the display unit 530 may include a touch screen 531 disposed on the front of the terminal device 110, which can collect touch operations on or near the object, such as clicking a button, dragging a scroll box, etc.

[0195] The touchscreen 531 can be placed on top of the display screen 532, or the touchscreen 531 and the display screen 532 can be integrated to realize the input and output functions of the terminal device 110. After integration, it can be referred to as a touch display screen. In this application, the display unit 530 can display the application and the corresponding operation steps.

[0196] Camera 540 can be used to capture still images, and objects can publish images captured by camera 540 through an application. There can be one or multiple cameras 540. An object generates an optical image through a lens, which is projected onto a photosensitive element. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to processor 580 to be converted into a digital image signal.

[0197] The terminal device may also include at least one sensor 550, such as an accelerometer 551, a proximity sensor 552, a fingerprint sensor 553, and a temperature sensor 554. The terminal device may also be equipped with other sensors such as a gyroscope, barometer, hygrometer, thermometer, infrared sensor, light sensor, and motion sensor.

[0198] Audio circuitry 560, speaker 561, and microphone 562 provide an audio interface between the device and terminal device 110. Audio circuitry 560 converts received audio data into electrical signals, which are then transmitted to speaker 561, where they are converted into sound signals for output. Terminal device 110 may also be equipped with volume buttons for adjusting the volume of the sound signal. Conversely, microphone 562 converts collected sound signals into electrical signals, which are then received by audio circuitry 560, converted into audio data, and output to communication component 510 for transmission to, for example, another terminal device 110, or to memory 520 for further processing.

[0199] The Bluetooth module 570 is used to interact with other Bluetooth devices that also have a Bluetooth module via the Bluetooth protocol. For example, a terminal device can establish a Bluetooth connection with a wearable electronic device (such as a smartwatch) that also has a Bluetooth module through the Bluetooth module 570, thereby exchanging data.

[0200] The processor 580 is the control center of the terminal device, connecting various parts of the terminal through various interfaces and lines. It executes various functions and processes data by running or executing software programs stored in the memory 520 and calling data stored in the memory 520. In some embodiments, the processor 580 may include one or more processing units; the processor 580 may also integrate an application processor and a baseband processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the baseband processor mainly handles wireless communication. It is understood that the baseband processor may not be integrated into the processor 580. In this application, the processor 580 can run the operating system, applications, user interface display and touch response, as well as the eye expression generation method of this embodiment. Furthermore, the processor 580 is coupled to the display unit 530.

[0201] In some possible implementations, various aspects of the eye expression generation method provided in this application can also be implemented as a program product, comprising a computer program that, when run on a robot, causes the robot to perform the steps of the eye expression generation method according to the various exemplary embodiments of this application described above. For example, a computer device can perform actions such as... Figure 2A The steps are shown in the figure.

[0202] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0203] The program product of the embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include a computer program, and may run on an electronic device. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with a command execution system, apparatus, or device.

[0204] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a readable computer program. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with a command execution system, apparatus, or device.

[0205] Computer programs contained on readable media may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0206] Computer programs for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The computer program can execute entirely on the user's computer device, partially on the user's computer device, as a standalone software package, partially on the user's computer device and partially on a remote computer device, or entirely on a remote computer device. In cases involving remote computer devices, the remote computer device can be connected to the user's computer device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer device (e.g., via the Internet using an Internet service provider).

[0207] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0208] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0209] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing a computer-usable computer program.

[0210] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce a mechanism for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0211] These computer program commands may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the commands stored in the computer-readable storage medium produce an article of manufacture including command means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0212] These computer program commands can also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing the commands executed on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0213] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0214] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for generating eye expressions, characterized in that, include: Obtain the previous state set of the target object in the previous frame, and the real-time drive set in the current frame; The preceding state set refers to the historical expression information of the target object's eye expression in the previous frame, and the real-time driving set refers to the real-time expression information used to drive the generation of the target object's eye expression in the current frame. Based on the eye gaze position obtained from the preceding state set and the real-time driving set, the outline shape of the target object's initial eye expression in the current frame is adjusted to obtain the candidate eye expression of the target object in the current frame. Based on dynamic behavior data and preset data priorities, the candidate eye expressions are adjusted to obtain the target eye expression of the target object in the current frame. The dynamic behavioral data refers to the data that triggers changes in the eye expressions of the target object.

2. The method as described in claim 1, characterized in that, The step of adjusting the contour shape of the initial eye expression of the target object in the current frame based on the eye gaze position obtained from the preceding state set and the real-time driving set, to obtain candidate eye expressions of the target object in the current frame, includes: Extract the eye gaze position of the target object in the previous frame from the preceding state set, and extract the eye gaze position of the target object in the current frame from the real-time drive set; Based on the extracted eye gaze positions of the two frames, the gaze position offset of the target object in the two adjacent frames is obtained; When the gaze position of the target object in two adjacent frames is offset by more than the position offset threshold, the first movement mode is adopted to adjust the outline shape of the initial eye expression of the target object in the current frame and obtain the candidate eye expression of the target object in the current frame. When the gaze position offset of the target object in two adjacent frames is not greater than the position offset threshold, the second movement mode is adopted to adjust the contour shape of the initial eye expression and obtain the candidate eye expression of the target object in the current frame.

3. The method as described in claim 2, characterized in that, The first movement mode is used to adjust the outline shape of the target object's initial eye expression in the current frame, including: Based on the duration of the gaze jump from the current frame to the trigger time of this gaze jump and the gaze position of the target object in the current frame, the positions of the control points of each pupil contour of the target object's initial eye expression in the current frame are adjusted. After confirming that the control points of each pupil contour have been adjusted, based on the jump duration and the gaze position of the target object in the current frame, the position of each orbital contour control point of the target object in the current frame is further adjusted.

4. The method as described in claim 2, characterized in that, The second movement mode is used to adjust the contour shape of the initial eye expression, including: Based on the duration of the gaze jump from the current frame to the trigger time of this gaze jump and the gaze position of the target object in the current frame, the positions of each pupil contour control point and each eye socket contour control point of the target object's initial eye expression in the current frame are adjusted.

5. The method as described in claim 1, characterized in that, The process of adjusting the candidate eye expressions based on dynamic behavior data and preset data priorities to obtain the target eye expression of the target object in the current frame includes: Based on the facial expression material information obtained from the real-time driving set, the candidate eye expressions are fused to obtain the eye expressions of the target object in the current frame. Based on the emotional state information obtained from the preceding state set and the real-time driving set, the eye expressions of the material are processed by emotion fusion to obtain the emotional eye expressions of the target object in the current frame. Based on the facial expression flag information obtained from the real-time driving set, the emotional eye expression is subjected to behavior superposition processing to obtain the target eye expression of the target object in the current frame.

6. The method as described in claim 5, characterized in that, The process of fusing candidate eye expressions based on the facial expression material information obtained from the real-time driving set to obtain the target object's eye expression material in the current frame includes: If the facial expression material information in the real-time drive set is a material playback instruction, the progress update is performed on the material playback progress of the previous frame in the preceding state set. When the material playback progress of the current frame indicates that the current frame is in the material playback stage, the facial expression material of the first frame and the candidate eye expression are processed by facial expression fusion to obtain the material eye expression of the target object in the current frame. If the facial expression material information in the real-time drive set is a material exit instruction, the material exit progress of the previous frame in the preceding state set is updated. When the material exit progress of the current frame indicates that the current frame is in the material exit stage, the last frame facial expression material and the candidate eye expression are processed by facial expression fusion to obtain the material eye expression of the target object in the current frame. If the expression material information in the real-time drive set is a material switching instruction, a new eye expression material is obtained, and the new first frame eye expression material and the candidate eye expression are processed by expression fusion to obtain the target object's material eye expression in the current frame.

7. The method as described in claim 6, characterized in that, The step of performing expression fusion processing on the first frame of the expression material package and the candidate eye expressions to obtain the target object's eye expression in the current frame includes: If the first frame of the expression material is not in a closed-eye state, then based on the playback progress of the material in the current frame, a first transition weight is obtained to control the transition speed curve, and based on the first transition weight, the first frame of the expression material and the candidate eye expression are fused to generate the target object's eye expression in the current frame. If the first frame of the expression material shows the eyes closed, then the candidate eye expression is replaced with the first frame of the expression material to obtain the target object's eye expression in the current frame.

8. The method as described in claim 6, characterized in that, The step of performing expression fusion processing on the last frame's expression material and the candidate eye expressions to obtain the target object's eye expressions in the current frame includes: Based on the material exit progress of the current frame, a second transition weight is obtained to control the transition speed curve; Based on the second transition weight, the last frame's expression material and the candidate eye expressions are fused to generate the target object's eye expression material in the current frame.

9. The method as described in claim 5, characterized in that, Also includes: If the expression material information in the real-time driver set is a material pause playback command, the fusion step is paused, and the outline shape of the candidate eye expression remains unchanged.

10. The method as described in claim 5, characterized in that, The process of performing emotion fusion processing on the eye expressions of the source material based on the emotional state information obtained from the preceding state set and the real-time driving set to obtain the emotional eye expressions of the target object in the current frame includes: Based on the emotion type and emotion intensity obtained from the preceding state set and the real-time driving set, the emotion switching result of the target object in two adjacent frames is obtained. When the emotion switching result of the target object in two adjacent frames indicates that the target object's emotion has changed drastically, the outline shape of the eye expression of the material is adjusted based on the target emotion type and target emotion intensity of the target object in the current frame to obtain the emotional eye expression of the target object in the current frame. When the emotion switching result indicates that the target object's emotion has not changed drastically, the outline shape adjustment of the eye expression of the material driven by blinking behavior is performed to obtain the emotional eye expression of the target object in the current frame.

11. The method as described in claim 10, characterized in that, The step of adjusting the outline shape of the eye contour of the target object based on the target emotion type and intensity in the current frame to obtain the emotional eye expression of the target object in the current frame includes: Obtain multiple sets of contour control point sequences corresponding to the target emotion type and intensity of the target object in the current frame; For two adjacent sets of contour control point sequences, perform the following operations respectively: perform interpolation processing on the two adjacent sets of contour control point sequences to obtain the interpolated control point sequence in the emotional transition state; Based on multiple sets of contour control point sequences and multiple sets of interpolation control point sequences, the contour shape of the eye expression of the material is adjusted multiple times in succession, and the eye expression obtained by the last adjustment is taken as the emotional eye expression of the target object in the current frame.

12. The method as described in claim 10, characterized in that, The step of adjusting the outline shape of the eye expression in the material based on blinking behavior to obtain the emotional eye expression of the target object in the current frame includes: When it is determined that the current frame is in the blinking phase, obtain the blinking keyframes and adjacent keyframes associated with the current frame during the blinking process; Interpolation processing is performed on the associated blink keyframes and the blink contour control points contained in adjacent keyframes to obtain the interpolation control point sequence in the blink transition state. Based on the interpolation control point sequence during the blink transition, the outline shape of the eye expression in the material is adjusted to obtain the emotional eye expression of the target object in the current frame.

13. The method as described in claim 10, characterized in that, The step of adjusting the outline shape of the eye expression in the material based on blinking behavior to obtain the emotional eye expression of the target object in the current frame includes: When it is determined that the eye gaze position of the target object changes drastically between two adjacent frames, perform the following operations on the eye expression of each clip of the target object in sequence: Based on the duration of the jump and the total duration of the jump in the current frame from the trigger time of the current gaze jump, the closure weight of each eye socket contour control point in the eye expression of the material is obtained. Based on the closure weight of each contour control point, the position of each eye socket contour control point in the material's eye expression is adjusted to obtain the emotional eye expression of the target object in the current frame.

14. The method as described in claim 2, characterized in that, If the real-time drive set fails to extract the eye gaze position of the target object in the current frame, the following defocus search steps are executed repeatedly until the eye gaze position of the target object in the current frame is found or the set search time is reached: Based on the generated multiple virtual attention parameters, the target object is controlled to move within a preset range of multiple pupil contour control points of the initial eye expression in the current frame; During the movement, multi-dimensional environmental information around the target object is extracted, and based on the extracted multi-dimensional environmental information, the actual attention parameters for the current frame are regenerated. Based on the actual attention parameters of the current frame, the eye gaze position of the target object in the current frame is obtained.

15. The method as described in claim 14, characterized in that, During the out-of-focus search, the following pupil movement steps are executed repeatedly until the eye fixation position of the target object in the current frame is obtained or the set search duration is reached: Taking the target object's eye gaze position in the previous frame as the starting point of loss, the contour control points of the initial eye expression are controlled to slide inertially along the last eye movement direction before loss. The system controls each contour control point to stop moving for a set duration, and adjusts the inertial reset of each contour control point to drive the pupil center point of the target object in the current frame back to the center position of the eye.

16. The method as described in claim 5, characterized in that, The process of performing behavioral overlay processing on the emotional eye expressions based on the expression flag information obtained from the real-time driving set to obtain the target eye expressions of the target object in the current frame includes: If the expression flag information obtained by the real-time drive set is a forced blink or entering a sleep state, the contour shape of the emotional eye expression is adjusted by blink behavior to obtain the target eye expression of the target object in the current frame.

17. The method as described in claim 12, 13, or 16, characterized in that, Also includes: Extract the remaining blink cool-down time from the blink cool-down timer in the preceding state set; If the blinking behavior trigger interval between two adjacent frames of the target object is less than the blinking cooldown time, the blinking cooldown mechanism is activated to prevent the superposition of blinking behavior in the current frame. If the blinking behavior trigger interval between two adjacent frames of the target object is not less than the blinking cooldown time, the blinking cooldown mechanism is deactivated, allowing the blinking behavior to be superimposed in the current frame, and the blinking cooldown timer is restarted to start a new round of cooldown timing.

18. The method as described in claim 1, characterized in that, Before acquiring the target object's preceding state set in the previous frame and the real-time driving set in the current frame, perform the following steps to construct the target object's standard eye contour: Construct multiple sets of contour control point sequences, one set of contour control point sequences is used to depict the contour of an eye component; For each set of contour control point sequences, the layer-by-layer interpolation step is performed iteratively until the current layer contour control point sequence contains only two contour control points: interpolation processing is performed based on the two adjacent contour control points of the current layer contour control point sequence and the corresponding chord length accumulation sequence to obtain the interpolated control points located between the two adjacent contour control points, and each interpolated control point generated in the current layer is used as the contour control point sequence of the next layer; the chord length accumulation sequence is used to represent the approximate cumulative arc length from each contour control point to the first contour control point in the target control point index table; Based on each set of contour control point sequences and each set of corresponding multi-layer interpolation control points, a smooth curve contour of each eye component is fitted to form the standard eye contour of the target object in the first frame.

19. The method as described in claim 18, characterized in that, Perform the following steps to generate the chord length accumulation sequence for the i-th contour control point: A target control point index table for a single-layer contour control point sequence is constructed using a modular loop approach. Based on the geometric distance between the i-th contour control point and the (i-1)-th contour control point in the target control point index table, and the cumulative chord length sequence of the first (i-1) contour control points, the cumulative chord length sequence of the i-th contour control point is generated, where i is a positive integer.

20. The method as described in claim 19, characterized in that, The method employs a modular loop approach to construct a target control point index table for a single-layer contour control point sequence, including: For each contour control point in the original control point index table, perform the following operations: Take the index value of the i-th contour control point in the original control point index table and perform a modulo operation with the index value of the last contour control point in the table to obtain a first modulo result; Perform another modulo operation with the first modulo result and the index value of the last contour control point in the table to determine a target contour control point that has a mapping relationship with the i-th contour control point; Use the index value of the target contour control point in the original control point index table as the new index value of the i-th contour control point. Based on the new index values ​​of all contour control points, construct the target control point index table.

21. An eye expression generation device, characterized in that, include: The state acquisition module is used to acquire the previous state set of the target object in the previous frame, as well as the real-time drive set in the current frame. The preceding state set refers to the historical expression information of the target object's eye expression in the previous frame, and the real-time driving set refers to the real-time expression information used to drive the generation of the target object's eye expression in the current frame. The shape adjustment module is used to adjust the outline shape of the initial eye expression of the target object in the current frame based on the eye gaze position obtained from the preceding state set and the real-time drive set, so as to obtain the candidate eye expression of the target object in the current frame. The behavior overlay module is used to adjust the eye expressions of the candidate eye expressions based on dynamic behavior data and preset data priorities, so as to obtain the target eye expression of the target object in the current frame. The dynamic behavioral data refers to the data that triggers changes in the eye expressions of the target object.

22. A robot, characterized in that, It includes a processor and a memory, wherein the memory stores program code that, when executed by the processor, causes the processor to perform the steps of the method according to any one of claims 1-20.

23. A computer-readable storage medium, characterized in that, It includes program code that, when run on a computer device, causes the computer device to perform the steps of the method according to any one of claims 1-20.

Images