A method and system for real-time generation of parametrically driven eye animation

Abstract

The application discloses a kind of parameterization driven eye animation real-time generation method and system, system includes expression configuration file setting module, affine transformation module, animation image generation module, animation image playing module, the setting of expression configuration file in method is sequentially executed;Selected expression and find the expression configuration file of selected expression, data in configuration file is stored in array, the data of one frame eye animation in array in memory is obtained, and is converted into the affine transformation matrix of layer, to establish the affine transformation relationship of original eye image coordinates to changed eye image coordinates, so that a group of parameters corresponds one frame animation;Based on affine transformation matrix, each layer of original eye image is transformed, and the eye image after one frame animation change is obtained after merging;According to the expression of each frame, the eye image after change is generated frame by frame to replace the previous frame eye image;The physical precision of eyeball change is improved, artistic expression is improved, and calculation and rendering overhead are saved.

Description

Technical Field

[0001] This invention belongs to the field of digital animation generation technology, specifically relating to a parametrically driven real-time eye animation generation method and system. Background Technology

[0002] In the fields of computer graphics and real-time animation, the eyes, as the core carrier of facial expressions, directly determine the realism of a character's performance. Traditional eye animation mostly employs keyframe hand-drawing or skeletal rigging techniques, which face three major technical bottlenecks:

[0003] 1. High layer coupling: Existing methods usually treat eye structures (such as iris and sclera) as a single entity, making it difficult to independently control details such as changes in light and shadow and pupil dilation;

[0004] 2. Insufficient parameter control dimensions: It can only be driven by basic parameters such as rotation / displacement, and cannot achieve dynamic effects that conform to biomechanics, such as horizontal scaling and gradual change of transparency.

[0005] 3. Limited real-time performance: The lack of transparency channel optimization when rendering multiple layers may lead to a waste of GPU resources. Summary of the Invention

[0006] To address the shortcomings of existing technologies and achieve the goal of reducing generation overhead and generating more realistic and detailed animated eyes, this invention adopts the following technical solution:

[0007] A parameter-driven real-time eye animation generation method includes the following steps:

[0008] Configure emoticon profiles;

[0009] Select an emoji and find its configuration file, then store the data from the configuration file into an array;

[0010] The data of the first frame of eye animation in the array in memory is obtained and converted into the affine transformation matrix of the layer to establish the affine transformation relationship from the original eye image coordinates to the transformed eye image coordinates, so that a set of parameters corresponds to one frame of animation. The parameters can be generated in advance and saved as a data table file for use as an expression library, or they can be generated in real time.

[0011] Based on the affine transformation matrix, each layer of the original eye image is transformed and then merged to obtain the eye image after one frame of animation transformation.

[0012] The system generates updated eye images frame by frame based on the facial expression in each frame, replacing the previous eye image.

[0013] Furthermore, based on the expression configuration file, the animation layer system is controlled. The layers of the eye animation, from front to back, include the sclera layer, iris layer, pupil layer, light and shadow layer, decoration layer, eyelid layer, and functional layer. Each layer covers the previous layer. The layers contain transparency information. Each layer is composed of sublayers, and each sublayer contains a set of image materials. The corresponding image materials are loaded into the video memory and / or system memory before the animation plays, instead of preloading all materials or loading the required image materials for each frame, thereby balancing the memory, CPU, and storage I / O load.

[0014] Furthermore, the data in the array consists of layer transformation parameters, including horizontal scaling, vertical scaling, horizontal offset, vertical offset, rotation angle, and transparency. The rotation angle is used to construct first matrix parameters, second matrix parameters, third matrix parameters, and fourth matrix parameters with the horizontal scaling, vertical scaling, horizontal offset, and vertical offset, respectively. The first matrix parameters, third matrix parameters, 0, and 0 form the first column of the affine transformation matrix. The second matrix parameters, fourth matrix parameters, 0, and 0 form the second column of the affine transformation matrix. The 0, 0, 1, and 0 form the third column of the affine transformation matrix. The horizontal offset, vertical offset, 0, and 1 form the fourth column of the affine transformation matrix. The transformed horizontal coordinate, vertical coordinate, 1, and 1 are multiplied by the affine transformation matrix. This multiplication is then combined with the original image horizontal coordinate, original image vertical coordinate, 1, and 1 to construct an equation, thus generating the affine transformation relationship.

[0015] Furthermore, the first matrix parameter is the product of the cosine of the counterclockwise rotation angle and the lateral expansion amount; the second matrix parameter is the product of the sine of the counterclockwise rotation angle and the negative value of the lateral expansion amount; the third matrix parameter is the product of the sine of the counterclockwise rotation angle and the longitudinal expansion amount; and the fourth matrix parameter is the product of the cosine of the counterclockwise rotation angle and the longitudinal expansion amount.

[0016] Furthermore, based on the eye's focal point, the lateral and longitudinal offsets of the focal point are accumulated for the lateral and longitudinal offsets of the sclera, iris, pupil, light and shadow layer, and decorative layer, respectively, thus giving the gaze a focal point.

[0017] Furthermore, by adjusting the lateral offset of the eye, the lateral bias, and the light and shadow rotation angle coefficient, the counterclockwise rotation angle of the light and shadow layer is adjusted so that the light and shadow layer rotates accordingly with the lateral offset of the eye, simulating the light and shadow effect of reflection moving on the eyeball as the eyeball moves.

[0018] Furthermore, a mood coefficient is constructed, and the mood coefficient is multiplied by the first, second, third, and fourth matrix parameters of the pupil layer, so that the pupil of the eye changes with the mood of the animated character.

[0019] Furthermore, based on the affine transformation matrix, the sub-layers of each layer are transformed, and the layers are merged according to the transformed layers and the transparency information to obtain the eye image after the original eye image has been transformed by one frame of animation.

[0020] Furthermore, the expression configuration file contains different expressions, and for the same expression, there is a set of expression configuration files. The generated eye animation is played frame by frame based on the current expression. If the current expression type changes, the remaining frames of the eye animation generated for the current expression are discarded, and a new eye animation is regenerated based on the data in the selected expression configuration file for the new expression. Otherwise, the playback of the original expression's eye animation is completed, and other expression animations are generated and played based on the expression configuration file.

[0021] A parameterized real-time eye animation generation system includes an expression profile setting module, an affine transformation module, an animation image generation module, and an animation image playback module. According to the parameterized real-time eye animation generation method, the system sequentially performs expression profile setting, selection, eye animation image generation, and eye animation playback.

[0022] The advantages and beneficial effects of this invention are as follows:

[0023] This invention employs a seven-layer parametric-driven architecture. By deconstructing visual elements of the eye into independent, controllable units such as the sclera and iris layers, and introducing a 6-dimensional parameter matrix, each layer supports horizontal / vertical non-uniform scaling to match the deformation patterns of the real eyeball, thus improving physical accuracy. The overlay mechanism of decorative and lighting layers supports micro-expression-level detail customization, enhancing artistic expressiveness. Layer compositing based on the alpha channel sets unnecessary parts of each layer to transparent and delegates the rendering to OpenGL ES, saving a significant amount of CPU decision-making and branch calculations, reducing real-time rendering overhead, and improving rendering efficiency. Animation generation through parametric driving actually generates parameters, which consumes far fewer computational resources and storage space than directly generating video, making it more suitable for embedded systems. Furthermore, the parameters can be pre-defined with functional relationships, making it easier to achieve specific effects and providing greater variety and less repetition. In addition, since the animation parameters themselves can be adjusted in real time, combined with technologies such as facial recognition and eye recognition, it can achieve effects such as eye contact and face tracking, and can react instantly to external stimuli, overcoming the shortcomings of traditional robots with lifeless eyes that can only play preset animations. This invention provides a high-fidelity eye animation solution for scenarios such as virtual reality and digital human interaction. Attached Figure Description

[0024] Figure 1 This is a flowchart of the method in an embodiment of the present invention.

[0025] Figure 2This is a schematic diagram of the functional layer in an embodiment of the present invention.

[0026] Figure 3 This is one of the schematic diagrams of the eyelid layer in the embodiments of the present invention.

[0027] Figure 4 This is the second schematic diagram of the eyelid layer in an embodiment of the present invention.

[0028] Figure 5 This is a schematic diagram of the decorative layer in an embodiment of the present invention.

[0029] Figure 6 This is one of the schematic diagrams of the light and shadow layer in the embodiments of the present invention.

[0030] Figure 7 This is the second schematic diagram of the light and shadow layer in the embodiments of the present invention.

[0031] Figure 8 This is the third schematic diagram of the light and shadow layer in the embodiments of the present invention.

[0032] Figure 9 This is a schematic diagram of the pupil layer in an embodiment of the present invention.

[0033] Figure 10 This is a schematic diagram of the iris layer in an embodiment of the present invention.

[0034] Figure 11 This is a schematic diagram of the sclera layer in an embodiment of the present invention.

[0035] Figure 12 This is a rendering of the eye animation synthesis in an embodiment of the present invention.

[0036] Figure 13 This is a schematic diagram of the system structure in an embodiment of the present invention. Detailed Implementation

[0037] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0038] like Figure 1 As shown, a parameterized-driven real-time eye animation generation method is implemented by an eye animation generation device. This device can be configured in various forms of computer systems, devices, and cloud environments, without specific limitations here. The parameters can be predefined manually or generated in real time through artificial intelligence algorithms. Here, we take predefined parameters as an example, including the following steps:

[0039] Step S1: Write the preset emoticons into a configuration file and store it in the flash module of the microcontroller system.

[0040] Among them, the animation layer system controlled by the preset expression configuration file, such as Figures 2 to 11 As shown, the eye animation consists of seven basic layers, from front to back: sclera layer, iris layer, pupil layer, lighting layer, decoration layer, eyelid layer, and functional layer. These seven basic layers are displayed sequentially, with each subsequent layer overlapping the previous one. Each layer contains transparency information. Each basic layer can be composed of multiple sub-layers. For example, the functional layer can be composed of battery indicator layers, network status indicator layers, weather status indicator layers, etc., and the eyelid indicator layers include upper and lower eyelid indicator layers. Each layer contains one or more image assets, which are loaded into video memory or system memory before the animation plays, eliminating the need to read them from external storage during playback. Loading the corresponding image assets before each animation begins, rather than preloading all assets or loading the required image assets in each frame, balances memory, CPU, and storage I / O load.

[0041] Step S2: Select an emoticon, find the configuration file for the selected emoticon in Flash, store the data in the configuration file into array a, and save the data into memory.

[0042] The selected "happy" emoji is triggered by external conditions, and the emoji is numbered 3 in the preset configuration file. The external trigger conditions can be that a person clicks on the interactive interface to select the happy emoji, or the artificial intelligence system determines that a happy emoji needs to be made when interacting with a human. That is, the trigger selection conditions, triggering methods and other methods are not specifically limited here.

[0043] Step S3: Read the data of array 'a' from the first frame of the animation in memory and convert it into an affine transformation matrix for each layer. The controllable layer transformation parameters include six parameters: horizontal scaling, vertical scaling, horizontal offset, vertical offset, rotation, and opacity. The horizontal scaling, vertical scaling, horizontal offset, vertical offset, and opacity parameters are dimensionless, with values ​​ranging from [0,1]; the rotation parameter ranges from [0°, 360°]. The process of performing affine transformations using these control parameters is as follows:

[0044]

[0045] Because the OpenGL standard uses a 4x4 matrix, although it is a 2D animation, the size of the transformation matrix is ​​still 4x4, and c=0, f=0, g=0, h=0, i=1; ,in, This refers to the lateral expansion / contraction amount. This refers to the longitudinal expansion / contraction. It is a lateral offset. For vertical offset, The angle is counterclockwise rotation. The coordinates of the original image are... These are the transformed coordinates.

[0046] Each set of parameters corresponds to one frame of animation. The parameters can be generated in advance and saved as a data table file, which can be called as an emoji library, or they can be generated in real time.

[0047] Step S4: Based on the basic parameters from Step S3, fine-tune the parameters according to other information from the eye animation. Adjust the lateral offset of all layers except the eyelid layer and the functional layer according to the focus of the eye movement. and longitudinal offset Accumulate the focus offset separately and That is, the transformation matrix becomes: Based on the left and right shift of the gaze, the light and shadow layer is rotated accordingly to simulate the light and shadow effect of reflections moving on the eyeball, i.e., the light and shadow layer. ,in The light and shadow rotation angle coefficient is adjusted according to factors such as eye size; that is, the transformation matrix of the light and shadow layer becomes: The happy expression exemplified in this embodiment can adjust pupil dilation based on a mood index, multiplying the horizontal and vertical expansion by a mood coefficient greater than 1. That is, for the pupil layer, the transformation matrix becomes .

[0048] Step S5: Based on the matrix, convert the data into pixel count and transparency parameters according to the hardware configuration, transform the image of each layer, and save it to the display swap area to form a merged display effect, such as... Figure 12 As shown.

[0049] Step S6: Update the swap area and display the final image on the screen to complete the playback of one frame. After each frame finishes playing, determine the state of the next frame. If the current expression continues, wait for the next frame to play, return to step S3, and generate the second frame image. If other interaction requests are inserted, discard the remaining frames of expression 3 (happy), immediately return to step S2, and start playing the corresponding animation. If all frames of the expression 3 (happy) animation have finished playing, i.e., the expression 3 (happy) animation is complete, return to step S2, and continue playing other expression animations according to the configuration.

[0050] like Figure 13 As shown, this invention also proposes a parameterized-driven real-time eye animation generation system, including an expression profile setting module, an affine transformation module, an animation image generation module, and an animation image playback module. According to the parameterized-driven real-time eye animation generation method, the expression profile is set, selected, the eye animation image is generated, and the eye animation is played in sequence.

[0051] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A parameter-driven real-time eye animation generation method, characterized in that... Includes the following steps: Configure emoticon profiles; Select an emoji and find its configuration file, then store the data from the configuration file into an array; The data of one frame of eye animation is obtained from the array in memory and converted into an affine transformation matrix of the layer to establish an affine transformation relationship from the original eye image coordinates to the transformed eye image coordinates, so that a set of parameters corresponds to one frame of animation. The data in the array are layer transformation parameters, including horizontal scaling, vertical scaling, horizontal offset, vertical offset, rotation angle, and transparency. The rotation angle is used to construct the first matrix parameter, the second matrix parameter, the third matrix parameter, and the fourth matrix parameter with the horizontal scaling, vertical scaling, horizontal offset, and vertical offset, respectively. The first matrix parameter, the third matrix parameter, 0, and 0 form the first column of the affine transformation matrix. The second matrix parameter, the fourth matrix parameter, 0, and 0 form the second column of the affine transformation matrix. The third column of the affine transformation matrix is ​​formed by 0, 0, 1, and 0. The fourth column of the affine transformation matrix is ​​formed by the horizontal offset, vertical offset, 0, and 1. The transformed horizontal coordinate, vertical coordinate, 1, and 1 are multiplied by the affine transformation matrix, and then multiplied by the original horizontal coordinate, original vertical coordinate, 1, and 1 to construct an equation to generate the affine transformation relationship. Based on affine transformation, each layer of the original eye image is transformed and then merged to obtain the eye image after one frame of animation. The system generates updated eye images frame by frame based on the facial expression in each frame, replacing the previous eye image.

2. The method for real-time generation of eye animation driven by parameterization according to claim 1, characterized in that: Based on the expression configuration file, the animation layer system is controlled. The layers of the eye animation, from front to back, include the sclera layer, iris layer, pupil layer, light and shadow layer, decoration layer, eyelid layer, and functional layer. Each layer covers the previous layer. The layers contain transparency information. Each layer is composed of sublayers. Each sublayer contains a set of image materials. The corresponding image materials are loaded into video memory and / or memory before the animation plays.

3. The method for real-time generation of parametrically driven eye animation according to claim 1, characterized in that: The first matrix parameter is the product of the cosine of the counterclockwise rotation angle and the lateral expansion / contraction amount; the second matrix parameter is the product of the sine of the counterclockwise rotation angle and the negative value of the lateral expansion / contraction amount; the third matrix parameter is the product of the sine of the counterclockwise rotation angle and the longitudinal expansion / contraction amount; and the fourth matrix parameter is the product of the cosine of the counterclockwise rotation angle and the longitudinal expansion / contraction amount.

4. The method for real-time generation of parametrically driven eye animation according to claim 1, characterized in that: Based on the eye's focal point, the lateral and longitudinal offsets of the focal point are accumulated for the sclera, iris, pupil, light and shadow layers, and decorative layers, respectively.

5. The method for real-time generation of parametrically driven eye animation according to claim 1, characterized in that: The rotation angle of the light and shadow layer is adjusted by the lateral offset of the eye, the lateral bias, and the light and shadow rotation angle coefficient, so that the light and shadow layer rotates accordingly with the lateral offset of the eye.

6. The method for real-time generation of parametrically driven eye animation according to claim 3, characterized in that: A mood coefficient is constructed by multiplying the mood coefficient by the first, second, third, and fourth matrix parameters of the pupil layer, respectively.

7. The method for real-time generation of eye animation driven by parameterization according to claim 2, characterized in that: Based on the affine transformation matrix, the sub-layers of each layer are transformed. The layers are then merged according to the transformed layers and the transparency information to obtain the eye image after the original eye image has been transformed by one frame of animation.

8. The method for real-time generation of eye animation driven by parameterization according to claim 1, characterized in that: The expression configuration file contains different expressions. For the same expression, there is a set of expression configuration files. The generated eye animation is played frame by frame based on the current expression. If the current expression type changes, the remaining frames of the eye animation generated for the current expression are discarded, and a new eye animation is generated based on the data in the selected expression configuration file for the new expression. Otherwise, the playback of the original expression eye animation is completed, and other expression animations are generated and played based on the expression configuration file.

9. A parameter-driven real-time eye animation generation system, comprising an expression profile setting module, an affine transformation module, an animation image generation module, and an animation image playback module, characterized in that: According to any one of claims 1 to 8, a parameterized-driven real-time eye animation generation method is used to sequentially set and select an expression configuration file, generate an eye animation image, and play the eye animation.

Images