Animation generation method and apparatus for avatar, and electronic device, computer program product and computer-readable storage medium

By extracting posture information from the video data of the entity object for three-dimensional reconstruction, virtual image animation is generated, which solves the problems of low efficiency and poor real-time performance of virtual image animation, and realizes efficient and low-latency animation generation in live broadcasts, games and other scenarios , widely applicable.

WO2024244666A1PCT designated stage expired Publication Date: 2024-12-05TENCENT TECHNOLOGY (SHENZHEN) CO LTD

Patent Information

Application Number
PCT/CN2024/084579
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-26
Filing Date
2024-03-28
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

In the prior art, virtual image animation generation efficiency is low and the real-time performance is poor, and it cannot meet the high real-time requirements of live broadcasts, games and other scenarios. It relies on expensive motion capture equipment, which limits its wide application.

Method used

By obtaining the video data of the entity object, extracting posture information, performing three-dimensional reconstruction, generating the action and expression data of the virtual image, combining appearance resources to synthesize animation data, real-time animation generation of the virtual image is achieved without the need for expensive motion capture equipment.

Benefits of technology

It realizes efficient and low-latency virtual image animation generation in live broadcasts, games and other scenarios, meeting the needs of high real-time, with high animation generation efficiency, low latency, and no dependence on expensive equipment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024084579_05122024_PF_FP_ABST
    Figure CN2024084579_05122024_PF_FP_ABST
Patent Text Reader

Abstract

The present application belongs to the technical field of computers. Disclosed are an animation generation method and apparatus for an avatar, and an electronic device, a computer program product and a computer-readable storage medium. The method comprises: acquiring video data of an entity object; extracting posture information of the entity object on the basis of the video data, wherein the posture information represents a limb posture and an expression posture that the entity object presents in the video data; performing three-dimensional reconstruction on the entity object on the basis of the posture information, so as to obtain motion data and expression data of an avatar, wherein the motion data represents a limb motion of the avatar, which limb motion is obtained by means of reconstruction based on the limb posture, and the expression data represents a facial expression of the avatar, which facial expression is obtained by means of reconstruction based on the expression posture; and synthesizing animation data of the avatar on the basis of an appearance resource, the motion data and the expression data of the avatar, wherein the animation data represents that the avatar wears the appearance resource, presents the facial expression and executes the limb motion.
Need to check novelty before this filing date? Find Prior Art

Description

Method, device, electronic device, computer program product, and computer-readable storage medium for generating animation of virtual image

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the Chinese patent application with application number 202310613703.0 and application date May 26, 2023, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into this application as a reference. Technical Field

[0003] The present application relates to the field of computer technology, and in particular to a method, device, electronic device, computer program product, and computer-readable storage medium for generating an animation of a virtual image. Background Art

[0004] With the development of computer technology, avatars are increasingly used in live streaming, film and television, animation, gaming, virtual social networking, and human-computer interaction. Accurately driving avatars to produce smooth animations is crucial to their rendering performance.

[0005] Currently, real people perform according to the script, and motion capture (i.e., action capture) equipment captures the real people's body movements and facial expressions, and then converts the captured data into 3D motion data and 3D expression data of the virtual image, driving the virtual image to perform body movements and facial expressions similar to those of real people at continuous moments.

[0006] The above motion capture-based animation generation method is limited by the high cost of motion capture equipment and is only applicable to professional film and television production. It cannot be extended to general scenarios such as live broadcasts and games, and the animation generation efficiency of virtual images is low.

[0007] Summary of the Invention

[0008] The embodiments of the present application provide a method, device, electronic device, computer program product and computer-readable storage medium for generating animation of a virtual image, which does not rely on expensive motion capture equipment, can be extended to apply to general scenarios such as live broadcasts and games, and can reconstruct animations in real time to meet higher real-time requirements, with low animation generation latency and high animation generation efficiency.

[0009] The technical solution of the embodiment of the present application is implemented as follows:

[0010] An embodiment of the present application provides a method for generating an animation of a virtual image, the method being executed by an electronic device, the method comprising:

[0011] Get the video data of the entity object;

[0012] Extracting posture information of the entity object based on the video data, where the posture information represents the body posture and facial expression of the entity object presented in the video data;

[0013] Based on the posture information, the physical object is three-dimensionally reconstructed to obtain motion data and expression data of the virtual image, wherein the motion data represents the body movements of the virtual image obtained based on the body posture reconstruction, and the expression data represents the facial expressions of the virtual image obtained based on the expression posture reconstruction;

[0014] Based on the appearance resource of the virtual image, the action data and the expression data, animation data of the virtual image is synthesized, and the animation data represents that the virtual image wears the appearance resource, presents the facial expression and performs the body movement.

[0015] An embodiment of the present application provides a device for generating an animation of a virtual image, the device comprising:

[0016] an acquisition module configured to acquire video data of an entity object;

[0017] an extraction module configured to extract posture information of the entity object based on the video data, wherein the posture information represents a body posture and facial expression posture of the entity object presented in the video data;

[0018] a reconstruction module configured to perform three-dimensional reconstruction of the physical object based on the posture information to obtain motion data and expression data of the virtual image, wherein the motion data represents the body motion of the virtual image obtained by reconstructing the body posture, and the expression data represents the facial expression of the virtual image obtained by reconstructing the expression posture;

[0019] A synthesis module is configured to synthesize animation data of the virtual image based on the appearance resources of the virtual image, the action data and the expression data, wherein the animation data represents that the virtual image wears the appearance resources, presents the facial expression and performs the body movement.

[0020] An embodiment of the present application provides an electronic device, which includes one or more processors and one or more memories, wherein the one or more memories store at least one computer-executable instruction, and the at least one computer-executable instruction is loaded and executed by the one or more processors to implement the animation generation method of the virtual image as described above.

[0021] An embodiment of the present application provides a computer-readable storage medium, which stores at least one computer-executable instruction. The at least one computer-executable instruction is loaded and executed by a processor to implement the above-mentioned virtual image animation generation method.

[0022] An embodiment of the present application provides a computer program product comprising one or more computer-executable instructions stored in a computer-readable storage medium. One or more processors of an electronic device can read the one or more computer-executable instructions from the computer-readable storage medium and execute the one or more computer-executable instructions, thereby enabling the electronic device to perform the aforementioned method for generating an animation of an avatar.

[0023] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0024] Through the video data of the physical object, the body posture and facial expression of the physical object in the plane coordinate system are extracted, and then the physical object is reconstructed in three dimensions to obtain the body movements and facial expressions of the virtual image in the spatial coordinate system. Then, combined with the appearance resources of the virtual image, the animation data of the virtual image is synthesized, so that the virtual image can simulate and emulate the body movements and facial expressions of the physical object under the drive of the video data. This animation generation process does not rely on expensive motion capture equipment, and can be extended to general scenarios such as live broadcasts and games. It can also reconstruct animations in real time to meet higher real-time requirements, with low animation generation latency and high animation generation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0026] FIG1 is a schematic diagram of an implementation environment of a method for generating an animation of a virtual image provided by an embodiment of the present application;

[0027] FIG2 is a flow chart of a method for generating an animation of a virtual image provided by an embodiment of the present application;

[0028] FIG3 is a flow chart of a method for generating an animation of a virtual image provided by an embodiment of the present application;

[0029] FIG4 is a schematic flow chart of a virtual image animation generation solution provided in an embodiment of the present application;

[0030] FIG5 is a schematic diagram of a video format conversion method according to an embodiment of the present invention;

[0031] FIG6 is a schematic diagram of a skeleton skinning method provided in an embodiment of the present application;

[0032] FIG7 is a schematic diagram of an animation driving process provided in an embodiment of the present application;

[0033] FIG8 is a principle flow chart of an animation generation solution provided in an embodiment of the present application;

[0034] FIG9 is a logic diagram of an animation generation solution provided in an embodiment of the present application;

[0035] FIG10 is a schematic structural diagram of a virtual image animation generation device provided in an embodiment of the present application;

[0036] FIG11 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0037] FIG12 is a schematic structural diagram of another electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0039] In this application, the terms "first", "second", etc. are used to distinguish identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there any limitation on the quantity and execution order.

[0040] In the present application, the term "at least one" means one or more, and the term "plurality" means two or more. For example, a plurality of skeletal components means two or more skeletal components.

[0041] In this application, the term "including at least one of A or B" refers to the following situations: including only A, including only B, and including both A and B.

[0042] The user-related information (including but not limited to the user's device information, personal information, behavioral information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application, when applied to specific products or technologies in the manner of the embodiments of this application, are all permitted, agreed, authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant information, data and signals must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the video data of the physical objects involved in this application are all obtained with full authorization.

[0043] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0044] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and smart transportation.

[0045] Enabling computers to hear, see, speak, and feel is the future direction of human-computer interaction. Computer vision (CV) technology is one of the most promising approaches to human-computer interaction. Computer vision is the study of how to enable machines to "see." Specifically, it involves using cameras and computers to replace the human eye in identifying and measuring objects, performing machine vision tasks such as image recognition and measurement. Further image processing is performed to transform the computer-generated images into images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / action recognition, three-dimensional (3D) technology, 3D object reconstruction, virtual reality (VR), augmented reality (AR), simultaneous localization and mapping, autonomous driving, and smart transportation.

[0046] With the research and progress of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, Internet of Vehicles, automatic driving, smart transportation, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0047] The solution provided in the embodiments of this application involves artificial intelligence computer vision technology, specifically involving the application of the above computer vision technology in producing three-dimensional animation of virtual images, which will be explained in detail in the following embodiments.

[0048] The following describes the terms used in the embodiments of the present application.

[0049] Avatar: An object that can move in a virtual world. Typically, a avatar is a virtual, anthropomorphic digital image in a virtual world, such as a virtual person, anime character, or a virtual character. The avatar can be a three-dimensional model (i.e., a three-dimensional stereo model). The three-dimensional model can be a three-dimensional character built based on three-dimensional human skeleton technology. In some embodiments, the avatar can also be implemented using a 2.5-dimensional or 2-dimensional model, which is not limited in the embodiments of the present application. Typically, a three-dimensional computer graphics software such as Miku Miku Dance, an engine such as Unity, or an engine such as UE4 can be used to create a 3D model of the avatar. Of course, a two-dimensional computer graphics software such as Live2D can also be used to create a 2D model of the avatar. The dimensions of the avatar are not specifically limited here.

[0050] Metaverse: Also known as the metaverse, metaphysical universe, supersensory space, and virtual space, it is a network of 3D virtual worlds focused on social connections. The metaverse involves a persistent and decentralized online three-dimensional virtual environment.

[0051] Digital Human: A virtual image created by 3D modeling of the human body using information science methods, achieving a realistic simulation of the human body. Alternatively, a digital human is a digitized humanoid created using digital technology that closely resembles a human. Digital Humans are widely used in video creation, live broadcasts, industry reporting, social entertainment, and voice prompts. For example, digital Humans can serve as virtual anchors and avatars. Digital Humans are also referred to as virtual humans or virtual digital humans.

[0052] Virtual anchors (VATs) are hosts who use avatars to contribute content to video sites, such as virtual YouTubers (VTubers) and virtual uploaders (VUPs). Typically, VATs use their own personalities and images to engage in various forms of human-computer interaction, including broadcasting, performances, live broadcasts, and conversations.

[0053] The person behind the scenes refers to the person who performs or controls the virtual anchor behind the scenes during a live broadcast. For example, with the help of sensors installed on the head and limbs of the person behind the scenes, the body movements and facial expressions of the person behind the scenes are captured through an optical motion capture system, and the motion data is synchronized to the virtual anchor. In this way, with the help of the real-time motion capture mechanism, real-time interaction between the virtual anchor and the audience watching the live broadcast can be achieved.

[0054] Motion capture (MoCap), also known as motion capture, involves placing sensors on key parts of a moving object or real person. The motion capture system captures the sensor positions and then processes the motion data using a computer to generate three-dimensional spatial coordinates. Once the motion data is recognized by the computer, it can be applied in animation, gait analysis, biomechanics, ergonomics, and other fields. Common motion capture equipment includes motion capture suits.

[0055] Interpolation: This is a motion estimation and compensation method that can extend the number of frames in an animation clip to make the animation more coherent when the number of frames is insufficient. For example, a new frame is inserted between every two frames of the original animation clip, using the new frame to fill in the intermediate changes in body movements or facial expressions between the two previous frames.

[0056] Game engines are pre-programmed, editable computer game systems or the core components of interactive real-time graphics applications. These systems provide game designers with the tools they need to easily and quickly create games without having to start from scratch. Game engines include the following systems: animation engine, rendering engine, physics engine, collision detection system, sound effects, scripting engine, artificial intelligence, networking engine, and scene management.

[0057] Unreal Engine 4 (UE4): An industry-leading, AAA-rated, next-generation game engine developed by EPIC. It's a complete game development platform for next-generation consoles and DirectX 9-based PCs, providing a wealth of core technologies, data generation tools, and infrastructure support required by game developers. The UE4 engine boasts high efficiency, versatility, and the ability to directly preview development results. Its programming features visual Blueprint programming and supports multiple platforms, including consoles, PCs, and mobile phones. It's applicable to game development, film and television production, animation, and other fields.

[0058] Editor: A visual operation tool of the UE4 engine that integrates UE4 functions into a visual interface to facilitate users to quickly edit scenes. It also integrates various tools and serves as a bridge for users to use the engine.

[0059] Plugins: In the UE4 engine, plugins are collections of code and data that developers can enable or disable on a per-project basis within the editor. Plugins can add runtime gameplay functionality, modify built-in engine functionality (or add new functionality), create new file types, and extend the editor's functionality with new menus, toolbar commands, and sub-modes. Many UE4 subsystems can be extended using plugins.

[0060] Rendering engine: In the field of graphics technology, a rendering engine is responsible for rendering a completed 3D model for a virtual avatar into a 2D image, preserving the 3D model's three-dimensional effect. Typically, this 3D model-to-2D image rendering process is driven by the rendering engine's GPU rendering pipeline, resulting in the visualization of the virtual avatar represented by the 3D model on the display screen.

[0061] Graphics Processing Unit (GPU): A dedicated chip used in modern personal computers, servers, mobile devices, game consoles, etc. specifically for graphics image processing.

[0062] Graphics Application Programming Interface (GAPI): The process of communication between the Central Processing Unit (CPU) and the GPU is carried out according to a specific standard GAPI. Mainstream GAPIs include OpenGL, OpenGL ES, Direct X, Metal, Vulkan, etc. GPU manufacturers will implement certain standardized interfaces when producing GPUs. When developing graphics, you can call the GPU according to the methods defined by the interface.

[0063] Draw Call (DC) command: A graphics API typically provides a class of DC commands that allow the CPU to instruct the GPU to perform rendering operations. For example, the DrawIndexedPrimitive command in DirectX and the glDrawElement command in OpenGL are all DC commands supported in the corresponding graphics API.

[0064] Rendering pipeline: The graphics rendering process that runs on the GPU. The image rendering process typically involves the following rendering pipelines: vertex shader, rasterizer, and pixel shader. By writing code in the shader, you can control the GPU to render the rendering components.

[0065] Vertex Shader (VS): An essential component of the GPU rendering pipeline, a graphics processing unit that adds 3D effects. VS is programmable, allowing developers to adjust effects using new instructions. Each vertex is defined by a data structure. Basic attributes of a vertex include its x, y, and z coordinates. Vertex attributes may also include color, initial path, material, and lighting characteristics. The program then calculates each vertex of the 3D model based on the code and outputs the results to the next stage.

[0066] Rasterizer: A non-codable link in the GPU rendering pipeline. The program automatically assembles the output of the vertex shader or geometry shader into triangles, and rasterizes them into discrete pixels according to the configuration, and outputs them to the pixel shader.

[0067] Pixel Shader (PS): Also implemented as Fragment Shader (FS), a required link in the GPU rendering pipeline. After the model vertices are transformed and rasterized, they need to be filled with color. The FS / PS filling algorithm is for each pixel on the screen: the program will perform shading calculations on the rasterized pixels according to the code, and output them to the frame buffer after testing, completing a rendering pipeline process.

[0068] A frame buffer is a memory buffer that contains data representing all the pixels in a complete game frame. It's used in computer systems to store images being synthesized or displayed. Alternatively, a frame buffer is a bitmap contained in a portion of random-access memory (RAM) that drives the computer's display. Modern graphics cards contain frame buffer circuitry that converts the bitmap in memory into a signal that can be displayed on the screen.

[0069] Z-buffer (also known as depth buffer): The memory used to store the depth information of all pixels in the frame buffer is called the z-buffer or depth buffer. When rendering objects in a three-dimensional virtual scene, the depth (i.e., z-coordinate) of each generated pixel is stored in the z-buffer. The z-buffer is usually organized into an xy two-dimensional array that stores the depth of each screen pixel. In the z-buffer, the points of multiple objects appearing at the same pixel need to be sorted by depth, and the GPU performs calculations based on the depth sorting recorded in the z-buffer to achieve the depth perception effect of closer objects occluding farther objects.

[0070] Color buffer: The memory used to store the color information of all pixels in the frame buffer is called the color buffer. When rendering objects in a 3D virtual scene, each point that passes the depth test is assembled into discrete pixels by the rasterizer. The color of each discrete pixel is stored in the color buffer. Depending on the color mode, the color vector of the pixel has different formats.

[0071] Texture Mapping (also known as UV Mapping): U and V are the horizontal and vertical coordinates of the image on the display screen, respectively, and generally range from 0 to 1. For example, the U coordinate represents the Uth pixel in the horizontal direction / image width, and the V coordinate represents the Vth pixel in the vertical direction / image height. UV coordinates (also known as texture coordinates) are the basis for mapping the UV texture of the virtual image onto the surface of the virtual image's 3D model. UV coordinates define the position of each pixel in the image. These pixels are interconnected with the surface vertices of the 3D model and determine which pixel on the image the surface texture is projected onto. UV mapping accurately maps each pixel in the image to the surface of the 3D model. The software performs smooth image interpolation in the gaps between points. This is known as UV mapping. Furthermore, to ensure that the UV texture of the 3D model is properly distributed on the two-dimensional canvas, the three-dimensional surface is properly tiled on the two-dimensional canvas. This process is called UV unwrapping.

[0072] Point Cloud: A point cloud is a collection of irregularly distributed discrete points in space that represent the spatial structure and surface properties of a 3D object or scene. Point clouds are categorized into different types based on different classification criteria. For example, they can be divided into dense point clouds and sparse point clouds based on their acquisition method, or static point clouds and dynamic point clouds based on their temporal sequence type.

[0073] Point Cloud Data: The geometric information and attribute information of each point in the point cloud together constitute point cloud data. Geometric information is also called three-dimensional position information. The geometric information of a point in the point cloud refers to the spatial coordinates (x, y, z) of the point, including the coordinate values ​​of the point in each coordinate axis direction of the three-dimensional coordinate system, for example, the coordinate value x in the X-axis direction, the coordinate value y in the Y-axis direction, and the coordinate value z in the Z-axis direction. The attribute information of a point in the point cloud includes at least one of the following: color information, material information, laser reflection intensity information (also called reflectivity); usually, each point in the point cloud has the same amount of attribute information. For example, each point in the point cloud has two kinds of attribute information: color information and laser reflection intensity. For another example, each point in the point cloud has three kinds of attribute information: color information, material information, and laser reflection intensity information.

[0074] Three-dimensional reconstruction (3D reconstruction): The creation of mathematical models suitable for computer representation and processing of three-dimensional objects is the basis for processing, manipulating, and analyzing their properties in a computer environment. It is also a key technology for creating virtual reality that represents the objective world in computers. For example, 3D reconstruction of an avatar involves reconstructing the avatar's three-dimensional model, which involves two dimensions: 3D skeletal reconstruction and 3D facial reconstruction.

[0075] Mesh: The fundamental unit in computer graphics is called a mesh, and a common mesh is a triangular facet mesh. For 3D models, since they are composed of polygons, and a complex polygon is actually composed of multiple triangular faces, the outer surface of a 3D model is composed of multiple connected triangular faces. In three-dimensional space, the points that make up these triangular faces and the collection of triangle edges are called meshes. The points on the triangular faces of a mesh are called vertices of the 3D model.

[0076] Animation: Recording the state of an object at a specific moment in time using timeframes, then switching between them in a specific sequence and at specific intervals. The principles of animation in all software are similar. In the Unity engine, the behavior of each avatar (also known as animation behavior) is controlled by its associated animation state machine (Animator Controller).

[0077] Skeleton component: abbreviated as "skeleton", is a concept abstracted by an animation algorithm. The physical meaning of the skeletal component of an avatar is analogous to the human skeleton. The skeletal component is used to imitate the human skeleton, thereby controlling the animation behavior of the avatar's three-dimensional model.

[0078] Skeletal animation: A type of model animation, distinct from vertex animation. Currently, there are two types of model animation: vertex animation and skeletal animation. In skeletal animation, a 3D model has a skeleton structure consisting of interconnected "bone components." Technicians pre-create animation resources, which control the positional transformations of these skeletal components, indirectly driving the positional transformations of the mesh vertices bound to them, thereby generating animation data for the 3D model. This is typically suitable for complex animations involving a large number of meshes, such as running and jumping avatars.

[0079] Skeleton Skinning: After selecting a skeleton component, you can specify which Mesh vertices of the 3D model are driven by the skeleton component and how the weights are driven.

[0080] Vertex Animation: It is a type of model animation, which is different from skeletal animation. There are currently two ways of model animation: vertex animation and skeletal animation. Vertex animation is also called per-vertex animation, that is, the three-dimensional model is operated on vertex by vertex in the vertex shader to produce animation effects. In this way, each frame of animation is actually a "snapshot" of the three-dimensional model presenting a specific posture. By interpolating frames between different animation frames, the animation engine can obtain smooth animation effects. Vertex animation stretches each triangle in the mesh of the three-dimensional model to produce more natural movements (or expressions). It is currently mainly divided into per-vertex animation Morph and deformation target animation Pose.

[0081] Per-vertex animation (Morph animation) involves animating the vertices of a mesh. This motion data is then exported to the game engine, telling the engine how to move the vertices at runtime. This technique can produce any conceivable mesh deformation. However, it is data-intensive, as information about the motion of each vertex over time must be stored. Therefore, it is rarely used in real-time games.

[0082] Morph Target Animation (Pose Animation): Used in some real-time engines, this method also involves having a technician move the vertices of a mesh, but only creating a relatively small number of fixed extreme poses (Blender Shapes). Animation can be generated by blending two or more poses at runtime. The position of each vertex is obtained by linearly interpolating the vertex positions at each extreme pose.

[0083] NEON instruction set: A 128-bit SIMD (Single Instruction Multiple Data) extension architecture for the ARM Cortex-A series processors.

[0084] Universal Serial Bus (USB) external camera: refers to an external camera connected through a USB interface and recognized by the hardware platform through the UVC protocol.

[0085] USB Video Class (UVC) protocol: Microsoft and several other device manufacturers jointly developed a protocol standard for USB video capture devices (such as USB external cameras). It has now become a USB standard. The UVC protocol is one of the device class specifications in the USB specification and is a unified data exchange specification for video devices used as USB interfaces.

[0086] The following describes the technical concepts of the embodiments of the present application.

[0087] With the rapid development of technologies such as three-dimensional modeling, virtual reality, augmented reality, and metaverse in the field of computer vision, virtual images are becoming more and more widely used in live broadcasting, film and television, animation, games, virtual social networking, human-computer interaction, and other aspects.

[0088] As users' visual requirements for image quality (such as image quality, clarity, and resolution) continue to rise, precisely driving the avatar to produce smooth animations is crucial for rendering performance, ensuring that the avatar appears lifelike and vivid during animation. Animation generation here encompasses two key aspects: generating the avatar's body movements and its facial expressions. These two elements combine to create the avatar's animated behavior.

[0089] For example, in live streaming scenarios, where a virtual character acts as the host, delivering announcements or conversations, animation is required to enhance the avatar's realistic rendering. Similarly, in video creation scenarios, such as creating virtual host videos and digital human videos, animation is also required to enhance the avatar's realistic rendering.

[0090] Typically, in the animation production of virtual images, the following types of animation generation methods are used:

[0091] 1. Human body drive: The camera captures the physical behavior characteristics of real people and drives the skeletal components of the virtual image's 3D model, allowing the 3D model to imitate real people's behavior.

[0092] The above human-body driving method requires the input of offline data or a pre-recorded video to analyze the real-life body behavior characteristics, and then further extract the position and rotation information of each joint of the virtual image's 3D model (collectively referred to as posture data). The posture data of each joint is then used to drive the virtual image to perform animation behavior. However, due to the limitations of computational complexity and computational complexity, the real-time performance is poor, and it cannot be adapted to the UE4 engine for high-quality animation generation. The animation generation efficiency is low and the real-time performance is poor, which cannot meet the high-real-time animation generation requirements.

[0093] 2. Digital human drive: The UE4 engine comes with a digital human system. In the digital human system, digital humans are pre-set into multiple digital human components. Each pre-set digital human component is driven by the pre-set digital human action data.

[0094] In the above digital human driving method, the UE4 engine's digital human system is native, so all replaceable digital human components are pre-configured. Technicians cannot customize the digital human appearance resources and can only select from the existing component library. In addition, the UE4 engine's digital human driving is also completed offline, and only some driving animations can be preset to control the 3D model to execute animation. The animation generation efficiency is low and the real-time performance is poor, which cannot meet the high real-time animation generation requirements.

[0095] 3. Skeletal animation: The main way to make the virtual image move in the game. Usually, technicians will create skeletons and skin binding, and then create keyframe animation data for the virtual image, so that the virtual image can move according to the predetermined behavior trajectory.

[0096] With the above skeletal animation method, technicians must pre-produce a series of keyframe animation data in order to control the virtual image to move according to the predetermined behavior trajectory. Because its animation generation process is also offline, the animation generation efficiency is low and the real-time performance is poor, which cannot meet the high real-time animation generation requirements.

[0097] 4. Real-time motion capture drive: A real person (or actor) wears a motion capture suit with full-body sensors. The real person performs actions according to the script content and script audio. The motion capture suit captures the real person's body movements and facial expressions, and reports them to the computer connected to the motion capture suit. The computer transfers the body movements and facial expressions of the human body to the 3D model of the virtual image, and obtains 3D motion data and 3D expression data of each part of the virtual image's body, driving the virtual image to perform body movements and facial expressions similar to real people at continuous moments.

[0098] Although the above motion capture driving methods can meet real-time requirements, the technical threshold for using motion capture equipment is high and it is expensive. It is generally used in professional film and television production and is difficult to serve ordinary users. It cannot be extended to general scenarios such as live broadcasts and games, and the animation generation efficiency of virtual images is low.

[0099] In view of this, an embodiment of the present application proposes a method for generating animation for a virtual image, which can open up the following animation generation link in a game engine such as UE4: video data of a physical object is captured from a USB external camera, an electronic device performs three-dimensional reconstruction of the physical object, and outputs reconstructed motion data and expression data. The motion data controls the body movements of the virtual image, and the expression data controls the facial expressions of the virtual image. Combined with the motion data and expression data, the 3D model of the virtual image can be driven in real time to perform skeletal movements and make expressions, thereby controlling the virtual image to simulate the physical object to perform animation behavior and achieve high-quality animation rendering of the virtual image.

[0100] The above animation generation process requires no human intervention. Machines can quickly and accurately generate animation data for avatars, achieving high efficiency and meeting high real-time requirements. Furthermore, it eliminates the need for expensive motion capture equipment; only conventional cameras are required for real-time animation generation and production. Therefore, this process is applicable to a variety of scenarios requiring avatar animation, such as live streaming, on-demand video streaming, gaming, and digital human video.

[0101] Further analysis shows that the USB external camera can be used to run the virtual image on many display devices that do not have built-in cameras. In addition, the skeletal components of the virtual image have the characteristics of high-freedom replacement and real-time drive, and are not limited to the limited digital human components in the component library of the UE4 native digital human system. This can be expanded to digital human animation-driven scenes of any body shape, any face shape, and any appearance. It is not limited to digital humans. Other virtual images involved in games, animation, and film and television can all achieve real-time and fast animation generation in this way.

[0102] The following describes the system architecture of the embodiment of the present application.

[0103] FIG1 is a schematic diagram of an implementation environment for a method for generating an animation of a virtual character provided in an embodiment of the present application. Referring to FIG1 , the implementation environment includes a terminal 101 and a server 102. Terminal 101 and server 102 are directly or indirectly connected via a wireless network or a wired network, which is not limited in this application.

[0104] An application supporting virtual images is installed on the terminal 101. The terminal 101 can realize functions such as animation generation of virtual images through the application. Of course, the application can also have other functions, such as game development functions, network social functions, video sharing functions, video submission functions, or chat functions. Among them, the application is a native application in the operating system of the terminal 101, or an application provided by a third party. For example, the application includes but is not limited to: a game engine, an animation engine, a 3D animation application, a live broadcast application, a short video application, an audio and video application, a game application, a social application, or other applications. As an example, the application is a UE4 engine or a Unity engine, etc., which is not limited in the embodiments of the present disclosure.

[0105] In some embodiments, the terminal 101 is a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart TV, a smart watch, a car terminal, etc., but is not limited thereto.

[0106] The server 102 provides background services for applications that support virtual images on the terminal 101. The server 102 can store three-dimensional models of multiple virtual images, wherein the three-dimensional models can be divided into 3D skeleton models and 3D facial models. Users can select personalized appearance resources for the virtual image according to their needs in the application. In addition, users can also pinch the face of the virtual image: adjust the facial features configuration parameters in the 3D facial model (such as eye distance, pupil distance, jaw width, philtrum length, etc.) according to their needs. The server 102 includes at least one of a single server, multiple servers, a cloud computing platform, or a virtualization center. In some embodiments, the server 102 is responsible for the main animation generation calculation work, and the terminal 101 is responsible for the secondary animation generation calculation work; or, the server 102 is responsible for the secondary animation generation calculation work, and the terminal 101 is responsible for the main animation generation calculation work; or, a distributed computing architecture is used between the server 102 and the terminal 101 for collaborative animation generation calculation.

[0107] In some embodiments, server 102 is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0108] The terminal 101 may generally refer to one of a plurality of terminals, and the embodiment of the present disclosure is illustrated only by taking the terminal 101 as an example. Those skilled in the art will appreciate that the number of the terminals may be more or less.

[0109] In some embodiments, during the offline animation generation process, a user connects a USB external camera to terminal 101 and authorizes the external camera to take photos. Terminal 101 then uses the USB external camera to capture video data of the user (an example of a physical object) and sends an animation generation request carrying the video data to server 102. Server 102 responds to the animation generation request and, based on the video data, utilizes the virtual image animation generation method of the embodiments of the present application to drive the generation of animation data for the virtual image, so that the virtual image's body movements and facial expressions are highly simulated to the body movements and facial expressions made by the user in the video data. Server 102 then returns the virtual image animation data to terminal 101, causing terminal 101 to play the virtual image animation data. In this way, terminal 101 does not need to connect to expensive motion capture equipment, and virtual image animation generation driven by video signals is achieved, with high animation generation efficiency.

[0110] In some embodiments, if the application locally installed on terminal 101 has an integrated animation generation function, terminal 101 does not need to request server 102 to return animation data. Instead, terminal 101 can locally generate animation data for the avatar using the video data and then play the animation data for the avatar. This saves terminal 101 the communication overhead of the animation generation process and eliminates the need to request server 102 to participate in the animation generation.

[0111] In some embodiments, during the real-time animation generation process, a user connects a USB external camera to a terminal 101 and authorizes the external camera to take pictures. The terminal 101 then uses the USB external camera to capture a video data stream of the user (an example of a physical object). The video data stream includes video frames captured at consecutive moments. The terminal 101 streams the video data stream to the server 102 to request the server 102 to drive the real-time animation generation. The server 102 continuously receives the video data stream pushed by the terminal 101 and, based on each video frame in the video data stream, drives the generation of an animation frame for the virtual image using the animation generation method for the virtual image according to the embodiment of the present application. In this way, multiple consecutive video frames drive the generation of multiple consecutive animation frames, thereby synthesizing the animation data stream of the virtual image. This allows the corresponding animation stream to be synthesized under the drive of the real-time video stream, and the body movements and facial expressions of the virtual image in the animation stream are controlled to be highly simulated to the body movements and facial expressions made by the user in the video stream. The server 102 then returns the animation data stream of the virtual image to the terminal 101, causing the terminal 101 to play the animation data stream of the virtual image. In this way, the terminal 101 side does not need to be connected to expensive motion capture equipment, and the virtual image animation generation driven by the video stream signal is realized. This method has a very low latency, can meet higher real-time requirements, and has high animation generation efficiency. It can be extended to scenarios with high real-time requirements such as live broadcasts and games.

[0112] In some embodiments, if the application locally installed on terminal 101 has an integrated animation generation function, terminal 101 does not need to request server 102 to return the animation data stream. Instead, terminal 101 can locally generate the animation data stream of the avatar using the video data stream, and then play the animation data stream of the avatar. This saves the communication overhead of the animation generation process on terminal 101, and does not require server 102 to participate in the animation generation. This can be applied to scenarios with high real-time requirements, such as live broadcasts and games.

[0113] The avatar animation generation method provided in the embodiments of this application can be applied to any scenario requiring avatar animation. For example, in a live broadcast scenario involving a digital human, the avatar does not need a motion capture suit to perform. All that is needed is a terminal with a camera, or a terminal without a camera but connected to an external USB camera. As long as the terminal has a camera function, after receiving authorization from the avatar, the terminal captures the avatar's video data stream (i.e., live video stream) through its built-in or external camera. Driven by the video data stream, the terminal, either locally or by requesting a server, synthesizes the avatar's animation data stream, thereby controlling the avatar to mimic the avatar's body movements and facial expressions. This, combined with the avatar's own audio and subtitles (or lack thereof), enhances the authenticity and fun of the avatar's live broadcast. For another example, the method can also be applied to various scenarios requiring avatar animation generation, such as digital human customer service, animation production, film and television special effects, digital human hosting, and digital human video. The embodiments of this application do not specifically limit these application scenarios.

[0114] The following describes the basic process of the method for generating an animation of a virtual image according to an embodiment of the present application.

[0115] Figure 2 is a flow chart of a method for generating an animation of a virtual image provided by an embodiment of the present application. Referring to Figure 2, this embodiment is executed by an electronic device, which can be a terminal or a server. Taking the electronic device as an example, this embodiment includes the following steps.

[0116] 201. The terminal obtains video data of the entity object.

[0117] Entity objects refer to physical, movable objects in the real world, such as real people, robots, and animals. This application does not specifically limit the types of entity objects. For example, in a digital human live broadcast scenario, the entity object refers to the person controlling the virtual anchor. The person behind the virtual anchor is the person performing or controlling the virtual anchor during the live broadcast.

[0118] The video data refers to video data or a video data stream containing a physical object. The video data may include at least one video image, and the video data stream may include multiple video frames. The embodiments of the present application do not specifically limit whether the video data is a video segment or a continuously collected video stream.

[0119] In some embodiments, the terminal itself has a built-in camera or an external camera. In this case, after the terminal requests the physical object for photography permission, the terminal uses the camera to capture video data of the physical object after the physical object fully authorizes it. As an example, the external camera is a USB external camera or an external camera that can be connected to the terminal through other interfaces.

[0120] In some embodiments, the terminal establishes a communication connection with an external image acquisition device. After obtaining full authorization from the entity object, the image acquisition device captures video data of the entity object and sends the captured video data to the terminal through the communication connection so that the terminal receives the video data. The embodiment of the present application does not specifically limit whether the terminal itself or the external image acquisition device captures the video data.

[0121] In some embodiments, the terminal reads the video data of the entity object from a local database, or the terminal downloads the video data of the entity object from a cloud database. The embodiment of the present application does not specifically limit the source of the video data.

[0122] 202. The terminal extracts posture information of the entity object based on the video data, where the posture information represents the body posture and facial expression of the entity object presented in the video data.

[0123] The posture information represents the body posture and facial expression of the entity object in the video data. Therefore, the posture information refers to the two-dimensional position of the entity object in the video data.

[0124] In some embodiments, the posture information includes at least one of skeletal posture information or facial posture information, wherein the skeletal posture information indicates a two-dimensional posture of skeletal key points of the entity object, and the facial posture information indicates a two-dimensional posture of facial key points of the entity object.

[0125] As an example, the above-mentioned two-dimensional pose indicates position information and posture information. Then, for any skeletal keypoint or facial keypoint, the two-dimensional pose uses the u coordinate and v coordinate in the plane coordinate system to represent the two-dimensional position information, and also uses the rotation angle in the u direction and the v direction to represent the two-dimensional posture information. The u coordinate refers to the horizontal coordinate in the two-dimensional video image (or video frame) in the video data, and the v coordinate refers to the vertical coordinate in the two-dimensional video image (or video frame) in the video data.

[0126] In some embodiments, when it is necessary to model body movements, the posture information includes the skeletal posture information. Therefore, based on the video data obtained in step 201, the skeletal posture information of the entity object is extracted. For example, the two-dimensional posture of multiple skeletal key points of the entity object is extracted, and the two-dimensional posture of the multiple skeletal key points is determined as the skeletal posture information. It should be noted that when the video data stream is obtained in step 201, the skeletal posture information of the entity object is extracted for each video frame in the video data stream in a similar manner.

[0127] In some embodiments, when facial expressions need to be modeled, the posture information includes the facial posture information. Therefore, based on the video data obtained in step 201, the facial posture information of the entity object is extracted. For example, the two-dimensional poses of multiple facial key points of the entity object are extracted, and the two-dimensional poses of the multiple facial key points are determined as the facial posture information. It should be noted that when a video data stream is obtained in step 201, the facial posture information of the entity object is extracted for each video frame in the video data stream in a similar manner.

[0128] In some embodiments, when it is necessary to model body movements and facial expressions, the posture information includes skeletal posture information and facial posture information. Therefore, based on the video data obtained in step 201, the skeletal posture information and facial posture information of the entity object are extracted. For example, the two-dimensional postures of multiple skeletal key points of the entity object are extracted, and the two-dimensional postures of the multiple skeletal key points are determined as the skeletal posture information. In addition, the two-dimensional postures of multiple facial key points of the entity object are extracted, and the two-dimensional postures of the multiple facial key points are determined as the facial posture information. It should be noted that when the video data stream is obtained in step 201, the skeletal posture information and facial posture information of the entity object are extracted for each video frame in the video data stream in a similar manner.

[0129] 203. The terminal performs three-dimensional reconstruction of the physical object based on the posture information to obtain motion data and expression data of the virtual image, wherein the motion data represents the body motion of the virtual image reconstructed based on the body posture, and the expression data represents the facial expression of the virtual image reconstructed based on the expression posture.

[0130] Among them, the motion data represents the three-dimensional limb posture of the virtual image obtained after three-dimensional reconstruction of the two-dimensional limb posture of the physical object. Therefore, the motion data refers to the three-dimensional skeletal posture of the virtual image simulated based on the physical object, and the three-dimensional skeletal posture includes the three-dimensional posture of multiple skeletal key points of the virtual image.

[0131] Among them, the expression data represents the three-dimensional expression posture of the virtual image obtained after three-dimensional reconstruction of the two-dimensional expression posture of the physical object. Therefore, the expression data refers to the three-dimensional facial posture of the virtual image simulated based on the physical object, and the three-dimensional facial posture includes the three-dimensional posture of multiple facial key points of the virtual image.

[0132] As an example, the above three-dimensional pose indicates position information and posture information. Then, for any skeletal key point or facial key point, six pose parameters can be used to characterize its three-dimensional pose, of which three pose parameters characterize the position coordinates (i.e., position information) in the x, y, and z three-dimensional spatial coordinate system, and the other three pose parameters characterize the rotation angle (i.e., posture information) in the x, y, and z three-dimensional spatial coordinate system. As an example, the above three rotation angles are collectively referred to as Euler angles. Euler angles include: pitch angle Pitch, yaw angle Yaw, and roll angle Roll. Pitch angle Pitch represents the angle of rotation around the x-axis, yaw angle Yaw represents the angle of rotation around the y-axis, and roll angle Roll represents the angle of rotation around the z-axis. For facial key points, figuratively speaking, pitch angle Pitch can be regarded as the angle of "nodding", yaw angle Yaw can be regarded as the angle of "shaking head", and roll angle Roll can be regarded as the angle of "tilting head / shaking head".

[0133] In some embodiments, when it is necessary to model limb movements, the posture information extracted in step 202 includes skeletal posture information. Therefore, based on the skeletal posture information extracted in step 202, the physical object is subjected to three-dimensional skeleton reconstruction to obtain motion data of the virtual image. For example, the skeletal posture information is provided as a two-dimensional posture of multiple skeletal key points, and then the motion data is provided as a three-dimensional posture of the multiple skeletal key points. In this way, for each skeletal key point, three-dimensional skeleton reconstruction refers to the process of reconstructing its three-dimensional posture based on the two-dimensional posture of the skeletal key point. In this case, if facial expression reconstruction is ignored, the expression data of the virtual image can be configured as a preset expression (such as expressionless), or an animation state machine can be used to control the expression data of the virtual image.

[0134] It should be noted that when the video data stream is obtained in step 201, the skeletal posture information of the physical object will be extracted frame by frame in step 202, thereby realizing three-dimensional skeleton reconstruction frame by frame and obtaining the action data of the virtual image in each animation frame.

[0135] In some embodiments, when facial expressions need to be modeled, the posture information extracted in step 202 includes facial posture information. Therefore, based on the facial posture information extracted in step 202, the physical object is subjected to three-dimensional facial reconstruction to obtain expression data of the virtual image. For example, the facial posture information is provided as a two-dimensional pose of multiple facial key points, and the expression data is provided as a three-dimensional pose of the multiple facial key points. In this way, for each facial key point, three-dimensional facial reconstruction refers to the process of reconstructing its three-dimensional pose based on the two-dimensional pose of the facial key point. In this case, if skeletal reconstruction is ignored, the motion data of the virtual image can be configured as a preset action (such as a standing action), or an animation state machine can be used to control the motion data of the virtual image.

[0136] It should be noted that when the video data stream is obtained in step 201, the facial posture information of the entity object will be extracted frame by frame in step 202, so as to realize three-dimensional facial reconstruction frame by frame and obtain the expression data of the virtual image in each animation frame.

[0137] In some embodiments, when it is necessary to model limb movements and facial expressions, the posture information extracted in step 202 includes skeletal posture information and facial posture information. Therefore, based on the skeletal posture information extracted in step 202, the entity object is subjected to three-dimensional skeletal reconstruction to obtain motion data of the virtual image. For example, the skeletal posture information is provided as a two-dimensional posture of multiple skeletal key points, and the motion data is provided as a three-dimensional posture of the multiple skeletal key points. In this way, for each skeletal key point, three-dimensional skeletal reconstruction refers to the process of reconstructing its three-dimensional posture according to the two-dimensional posture of the skeletal key point; in addition, based on the facial posture information extracted in step 202, the entity object is subjected to three-dimensional facial reconstruction to obtain expression data of the virtual image. For example, the facial posture information is provided as a two-dimensional posture of multiple facial key points, and the expression data is provided as a three-dimensional posture of the multiple facial key points. In this way, for each facial key point, three-dimensional facial reconstruction refers to the process of reconstructing its three-dimensional posture according to the two-dimensional posture of the facial key point.

[0138] In some embodiments, when the video data stream is obtained in step 201, the skeletal posture information and facial posture information of the physical object are extracted frame by frame in step 202, thereby realizing three-dimensional skeletal reconstruction and three-dimensional facial reconstruction frame by frame, and obtaining the motion data and expression data of the virtual image in each animation frame.

[0139] 204. The terminal synthesizes animation data of the avatar based on the appearance resource of the avatar, the action data, and the expression data. The animation data represents that the avatar wears the appearance resource, presents the facial expression, and performs the body movement.

[0140] Appearance resources are used to control the external appearance of the avatar. For example, these appearance resources include, but are not limited to, hair, skin, eyes, clothing, accessories, special effects, etc. The animation data can be rendered to visually present an image of the avatar wearing these appearance resources, displaying these facial expressions, and performing these body movements.

[0141] In some embodiments, based on the image identification (ID) of the virtual image, the appearance resources of the virtual image are queried, and the queried appearance resources are combined with the action data and expression data reconstructed in step 203 to synthesize the animation data of the virtual image, so that the virtual image wears its own appearance resources and presents body movements and facial expressions that simulate the physical object.

[0142] It should be noted that when the video data stream is obtained in step 201, the skeletal posture information and facial posture information of the physical object will be extracted frame by frame in step 202, and the motion data and expression data of the virtual image will be reconstructed frame by frame in step 203. In this way, the animation data of the virtual image can be synthesized frame by frame in step 204, that is, for each video frame in the video data stream, an animation frame can be synthesized so that the body movements and facial expressions of the virtual image in the animation frame simulate the body movements and facial expressions of the physical object in the video frame.

[0143] As an example, when rendering the animation data of avatar, according to the image ID of avatar, the three-dimensional model stored in the form of point cloud can be queried, the three-dimensional model in the form of point cloud is baked, the grid data (i.e. Mesh) of the three-dimensional model can be derived, then each skeleton component of avatar is bound to the corresponding part in the grid data respectively, and the skeleton skinning of the three-dimensional model is realized. Because this three-dimensional model comprises a 3D skeleton model and a 3D facial model, the 3D skeleton model is split into multiple skeleton components, then the 3D facial model is regarded as a special skeleton component for controlling facial expression movements, so that the body movements and facial expressions of avatar can be controlled by all skeleton components. Further, the motion data in step 203 will control each skeleton component that the 3D skeleton model comprises to present certain body movements, and the expression data in step 204 will control the specific skeleton component that the 3D facial model comprises to present certain facial expressions. Furthermore, after the skeleton is skinned, the skeleton components are affected by the motion data or expression data, and when displacement or rotation occurs (i.e., posture changes), it will naturally affect the vertex position of the vertices in the bound mesh data. Therefore, the vertex position of each vertex in the mesh data needs to be recalculated. This is equivalent to applying deformation to the set vertices on the mesh data through motion data and expression data, thereby controlling the position offset of the vertices in the skeleton skin, so that the body movements and expression postures presented by the virtual image can be controlled from the vertex level, and are not limited to certain pre-set fixed animation behaviors. Therefore, the reconstruction accuracy of body movements and facial expressions is higher, and the degree of restoration and realism are higher.

[0144] Furthermore, based on the recalculated vertex position mesh data, combined with the avatar's own appearance resources, the GPU's rendering pipeline is driven to perform a series of rendering processes, including vertex shading, rasterization, and pixel shading, on the 3D model's mesh data. Ultimately, the avatar's animation data can be visualized, i.e., the avatar's animation data is drawn onto the terminal's display screen, thereby displaying the avatar's animation data. During the real-time animation generation process, simply displaying each animation frame of the avatar one by one can achieve the effect of playing the avatar's animation, realizing real-time, dynamic animation drive.

[0145] In some embodiments, a RHI (Render Hardware Interface) thread, namely RHIThread, is created in the terminal, and the appearance resources and deformed mesh data of the above-mentioned virtual image are submitted to the RHIThread. The RHIThread then executes the drawing commands in the graphics API, drives the rendering pipeline on the GPU to render the virtual image, and obtains an animation frame of the virtual image.

[0146] Below, the GPU rendering process of any animation frame will be explained. The GPU rendering pipeline involves vertex shader, rasterizer and pixel shader. Among them, the vertex shader is a rendering pipeline used to calculate the vertices of the mesh, and the rasterizer is a rendering pipeline used to assemble the results of the vertex shader output into a triangle mesh and rasterize it into discrete pixels according to the configuration. The pixel shader is a rendering pipeline used to perform shading calculations on each discrete pixel after rasterization and complete pixel shading.

[0147] For each vertex in the mesh data, the action and expression data control the vertex position. Of course, posture information such as rotation angle may also affect the vertex's depth value, while the appearance resource affects the vertex's color and depth values. First, the frame buffer's color and depth are cleared. Next, the vertex shader writes the depth of each vertex in the mesh data to the z-buffer, writing the depth value of each vertex to the z-buffer. This also involves a depth sorting process, which affects transparency and handles occlusion or translucency. Next, the rasterizer performs rasterization. Next, the pixel shader writes the color of each discretized pixel to the color buffer, writing the pixel value of each pixel to the color buffer. Each pixel value is calculated by integrating the color values ​​of all vertices located at that pixel. This color integration process typically takes into account vertex transparency, ultimately resulting in the output of a single animation frame of the avatar on the terminal's display. It's also important to note that the vertex shading stage often involves lighting calculations, with the specific lighting model used being determined by the avatar or rendering engine.

[0148] Among them, the frame buffer is used to store the data of all pixels in the current animation frame. The frame buffer contains a z buffer and a color buffer. The z buffer refers to the depth buffer in the frame buffer, which is used to store the depth value of each pixel in the current animation frame, and the color buffer is used to store the color value of each pixel in the current animation frame.

[0149] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0150] The method provided in the embodiment of the present application extracts the body posture and facial expression of the physical object in a plane coordinate system through the video data of the physical object, and then performs three-dimensional reconstruction of the physical object to obtain the body movements and facial expressions of the virtual image in the spatial coordinate system, and then combines the appearance resources of the virtual image to synthesize the animation data of the virtual image, so that the virtual image can simulate and emulate the body movements and facial expressions of the physical object under the drive of the video data. This animation generation process does not rely on expensive motion capture equipment, can be extended to general scenarios such as live broadcasts and games, and can reconstruct animations in real time to meet higher real-time requirements. The animation generation has low latency and high efficiency.

[0151] Furthermore, since animation rendering does not rely on skeletal animation or UE4's native digital human system, the synthesized virtual image animation is not limited to a few pre-set action categories, and there will be no problems such as mechanical feeling or rigid connection during animation presentation. The animation control of the virtual image can be achieved vertex by vertex, and the reconstruction accuracy of its body movements and facial expressions is detailed to the vertex level in the mesh data of the virtual image. It can control the virtual image to accurately imitate the body movements and facial expressions of the physical object with high precision, thereby improving the animation generation accuracy of the virtual image, having high flexibility and controllability, and optimizing the animation effect of the virtual image.

[0152] As shown in Figure 3, Figure 3 is a flowchart of a method for generating an animation of a virtual image provided in an embodiment of the present application. This embodiment is executed by an electronic device, which may be a terminal or a server. Taking the electronic device as a terminal as an example, this embodiment includes the following steps.

[0153] 301. The terminal captures video data of a physical object based on an external camera.

[0154] In some embodiments, when the terminal is equipped with an external camera, after requesting photography permission from the physical object, the external camera captures video data of the physical object after the physical object fully authorizes the physical object. As an example, the external camera is a USB external camera or an external camera that can be connected to the terminal through other interfaces.

[0155] In step 301, only the example of shooting video data through an external camera is used for explanation. In some embodiments, if the terminal itself is equipped with a built-in camera, the built-in camera can also be used to shoot video data. The embodiment of the present application does not specifically limit the camera type.

[0156] In some embodiments, in addition to capturing video data through a built-in camera or an external camera, the terminal can also establish a communication connection with an external image acquisition device. After obtaining full authorization from the entity object, the image acquisition device captures video data of the entity object and sends the captured video data to the terminal through the communication connection so that the terminal receives the video data. The embodiments of the present application do not specifically limit whether the terminal itself captures the video data or an external image acquisition device captures the video data.

[0157] In other embodiments, in addition to shooting video data, the terminal can also read the video data of the physical object from the local database, or the terminal can also download the video data of the physical object from the cloud database. The embodiments of the present application do not specifically limit the source of the video data.

[0158] It should be noted that the video data captured in step 301 refers to video data or a video data stream containing a physical object. The video data may include at least one video image, and the video data stream may include multiple video frames. This embodiment of the application does not specifically limit whether the video data is a single video segment or a continuously captured video stream.

[0159] As an example, taking the scenario of shooting video data stream as an example, the external camera refers to the USE external camera, which has lower hardware requirements for the terminal. The terminal only needs to have a USB slot to implement the animation generation solution of this application. After the USB external camera is inserted into the USB slot of the terminal (physical connection), the access configuration of the USB external camera (driving video) is configured to establish a communication connection between the USB external camera and the terminal. In this way, after requesting the shooting permission of the entity object, the terminal can call the camera interface (Application Programming Interface, API) through the operating system (Operating System, OS), drive the USB external camera through the camera interface to shoot the video data stream of the entity object in real time, and stream the shot video data stream to the terminal. In this way, as long as the USB external camera is encapsulated with call logic suitable for different operating systems, real-time video data acquisition can be completed on the terminal of any platform. Moreover, the user has no perception of the underlying call. He only needs to authorize the shooting permission to call the camera interface to drive the USB external camera to complete the shooting and obtain the video data stream returned by the USB external camera.

[0160] FIG4 is a schematic flow chart of a virtual image animation generation scheme provided by an embodiment of the present application. As shown in FIG4 , for a USB external camera, in order to ensure that the USB external camera and the terminal can complete the connection, both the USB external camera and the terminal need to support the UVC protocol, so that the terminal can recognize the connection of the USB external camera, and then the terminal and the USB external camera can achieve driver adaptation of each platform through the UVC protocol. After the physical connection and driver adaptation are completed, the terminal can drive the USB external camera to shoot the video data stream by calling the camera interface, and the USB external camera can return the captured video data stream to the terminal. Among them, the platform (or operating system) to which the terminal belongs includes but is not limited to: Android, Apple (iOS), Windows, Mac, etc. The embodiment of the present application does not specifically limit the device type, platform type, and operating system type of the terminal.

[0161] In some embodiments, a USB external camera is encapsulated into a unified camera interface at each platform layer. This allows the terminal operating system to seamlessly call the camera interface, eliminating the need to worry about which camera interface the platform itself calls to drive the USB external camera, thereby improving video data acquisition efficiency. Of course, the USB external camera can also be encapsulated into different camera interfaces for each platform layer. This way, when driving the USB external camera, the operating system needs to select the appropriate camera interface based on the platform to which it belongs. This is not specifically limited in the present embodiments.

[0162] In other embodiments, before driving the USB external camera to shoot the video data stream, the user can also configure the shooting performance of the USB external camera through the terminal, such as configuring its acquisition resolution, using a telephoto lens / short focal lens for acquisition, whether the camera has an anti-shake function turned on, etc. This allows for flexible configuration of the shooting performance of the USB external camera, which helps to capture higher quality video data streams.

[0163] 302. The terminal converts the video data from a video format supported by the external camera into a preset video format.

[0164] The preset video format is a format that supports three-dimensional reconstruction of physical objects.

[0165] In some embodiments, since the video data captured by different external cameras may have different video formats, in order to improve the efficiency of three-dimensional reconstruction, the video data captured by the external camera can be uniformly converted from the original video format to a preset video format, so as to facilitate the subsequent three-dimensional reconstruction of the video data in the preset video format. Step 302 can also be called a data preprocessing process. Step 302 is an optional step. If the video format supported by the external camera itself is a format that supports three-dimensional reconstruction of physical objects, then there is no need to expend computing power for video format conversion. This embodiment of the present application does not specifically limit this. Among them, the video format mainly refers to the data format of the video frame or video image (or the picture format, image format).

[0166] In the above steps 301 to 302, a possible implementation method for the terminal to obtain video data of the physical object is provided. By calling an external camera to shoot video data and then uniformly converting it into a preset video format, it can be applied to scenarios where real-time video streams are shot to drive real-time animation streams, and can meet high real-time business needs. It can also be applied to scenarios where offline video is recorded to drive animations. The preset video format is well compatible with the three-dimensional reconstruction algorithm, thereby improving the efficiency of three-dimensional reconstruction. In addition, there is no need to configure expensive motion capture equipment, and the cost is low, suitable for promotion and application, with high versatility and applicability.

[0167] As an example, FIG5 is a schematic diagram of a video format conversion method provided by an embodiment of the present application. As shown in FIG5 , if the external camera is a USB external camera, then since USB external cameras employ a variety of video encoding methods, the video data captured may also have a variety of video formats. During the data preprocessing stage, the video data in various video formats are converted to a preset video format. The preset video format refers to a data format supported by the 3D reconstruction algorithm. In some embodiments, hardware acceleration can be enabled during the video format conversion to reduce the latency of the video format conversion and improve the real-time performance of animation generation, but hardware acceleration is not a required step.

[0168] In some embodiments, the video format includes but is not limited to: YUV420, YUV444, RGBA, RGB, etc., among which YUV is a color encoding method that decomposes pixel values ​​into three channel components: Y (Luminance), U (Chrominance), and V (Chroma). Both YUV420 and YUV444 use the YUV color encoding method but have different sampling methods. YUV444 refers to 4:4:4 sampling, so that each Y corresponds to a group of UV components, and YUV420 refers to 4:2:0 sampling, so that every four Ys share a group of UV components; RGB is another color encoding method that decomposes pixel values ​​into three channel components: R (Red), G (Green), and B (Blue), and RGBA refers to adding an additional transparency channel Alpha on the basis of the three RGB channels.

[0169] As an example, assuming the preset video format is RGB or RGBA, then if the video data captured by a USB external camera is YUV420 or YUV444, then the video data needs to be converted from YUV420 or YUV444 to RGB or RGBA, which is equivalent to mapping from the YUV color encoding method to the RGB color encoding method, that is, converting the non-RGB color format into the RGB color format.

[0170] As an example, assuming the preset video format is YUV420 or YUV444, then if the video data captured by a USB external camera is RGB or RGBA, then the video data needs to be converted from RGB or RGBA to YUV420 or YUV444, which is equivalent to mapping the RGB color encoding method to the YUV color encoding method, that is, converting the RGB color format to a non-RGB color format.

[0171] In some embodiments, taking the conversion of the YUV color encoding method to the RGB color encoding method as an example, when converting the video format, it is necessary to access the video data through the CPU logic layer, read the pixel value pixel by pixel, and then input the source encoding format pixel value through the conversion logic function, and output the target encoding format pixel value. In this example, the source encoding format refers to the YUV color encoding method, and the target encoding format refers to the RGB color encoding method, wherein the conversion logic function refers to a function used to convert the pixel value from the YUV color encoding method to the RGB color encoding method.

[0172] Since the terminal will create a main thread (such as animation thread, rendering thread, etc.) for animation generation, the video format conversion method in the above example will block the terminal's current animation generation main thread until all pixels have completed the color coding method conversion. The subsequent logic of the main thread will not be executed. Therefore, the video format conversion process may cause a certain animation generation delay.

[0173] In view of this, in other embodiments, video format conversion can be accelerated to reduce the latency of this process and ensure that animation generation can meet high real-time requirements. It should be noted that the acceleration operation is an optional step in the data preprocessing stage, and video format conversion acceleration is not required. The following describes two possible ways to accelerate video format conversion.

[0174] Method 1: Sub-thread parallel acceleration

[0175] In some embodiments, a sub-thread for format conversion can be started, and the video data is converted from the video format supported by the external camera to the preset video format through the sub-thread. In other words, during the animation generation process, in order to avoid the main thread being blocked due to video format conversion, a sub-thread start instruction can be executed when the video format conversion is turned on to start a sub-thread. The sub-thread runs independently to complete the video format conversion, and the main thread continues to run subsequent logic without blocking. After the sub-thread completes the video format conversion of all pixels, it notifies the main thread that the video format conversion is complete, which is equivalent to providing an acceleration method for parallel processing of the main thread and the sub-thread, which can reduce the animation generation delay and improve the real-time performance of the animation generation. In some embodiments, the above-mentioned sub-thread start instruction can be a NEON instruction.

[0176] In other embodiments, similar to the parallel acceleration of the sub-threads, a coroutine for format conversion can also be started to convert the video data into the preset video format through the coroutine, which will not be repeated here.

[0177] Method 2: GPU hardware acceleration

[0178] In some embodiments, the drawing command of the graphics processor GPU can be called, and the video data is converted from the video format supported by the external camera to the preset video format through the GPU. As an example, calling the drawing command provided by the graphics API supported by the GPU can input the captured video data from the CPU to the GPU, and then the shader code of the GPU rendering pipeline can complete the video format conversion of each pixel in the GPU. Finally, the CPU can read the GPU texture and return the converted video data in the preset video format to the CPU. In other words, during the animation generation process, the GPU is used to perform video format conversion to achieve GPU hardware acceleration. In this way, the video format conversion is migrated from the CPU to the GPU, and the GPU itself is suitable for large-scale parallel computing at the hardware level. The computing speed is several orders of magnitude higher than that of the CPU. Therefore, hardware acceleration can greatly improve the efficiency of video format conversion, reduce animation generation delay, and improve the real-time performance of animation generation.

[0179] 303. The terminal determines the skeleton key points and facial key points of the entity object.

[0180] In some embodiments, when performing three-dimensional reconstruction of a physical object, a human body parametric model can be used to perform three-dimensional reconstruction of the human body. In this way, only a set of low-dimensional vectors is needed to describe the human body shape. In other words, only the two-dimensional pose of the skeletal key points and facial key points need to be extracted to describe the limb posture and facial expression posture of the physical object in the two-dimensional plane coordinate system. This key point-based three-dimensional reconstruction method has low computational complexity and high three-dimensional reconstruction efficiency. In some embodiments, the above-mentioned human body parametric model can be a skinned multi-person linear (SMPL) model, SMPL-X model, SCAPE model, etc. The embodiments of this application do not specifically limit the human body parametric model.

[0181] In the above parametric three-dimensional reconstruction process, when the human body parametric model adopted is different, the division method of bone key points and facial key points may also be different. Therefore, based on the human body parametric model adopted for three-dimensional reconstruction, multiple bone key points and multiple facial key points can be determined.

[0182] 304. The terminal extracts the skeletal posture information of the skeletal key points and the facial posture information of the facial key points based on the video data, and combines the skeletal posture information and the facial posture information into posture information of the entity object.

[0183] The skeletal posture information represents the two-dimensional position and posture of the skeletal key points, and the facial posture information represents the two-dimensional position and posture of the facial key points.

[0184] As an example, the above-mentioned two-dimensional pose indicates position information and posture information. Then, for any skeletal keypoint or facial keypoint, the two-dimensional pose uses the u coordinate and v coordinate in the plane coordinate system to represent the two-dimensional position information, and also uses the rotation angle in the u direction and the v direction to represent the two-dimensional posture information. Among them, the u coordinate refers to the horizontal coordinate in the two-dimensional video image (or video frame) in the video data, and the v coordinate refers to the vertical coordinate in the two-dimensional video image (or video frame) in the video data.

[0185] In some embodiments, since the video data itself contains the two-dimensional poses of key points in a two-dimensional plane coordinate system, based on the video data, the two-dimensional pose of each skeletal key point in the two-dimensional plane coordinate system can be extracted, and the two-dimensional pose of each skeletal key point in step 303 is used as the skeletal pose information of the entity object. Similarly, the two-dimensional pose of each facial key point in the two-dimensional plane coordinate system can be extracted, and the two-dimensional pose of each facial key point in step 303 is used as the facial pose information of the entity object. It should be noted that when the video data stream is obtained in step 301, the skeletal pose information and facial pose information of the entity object are extracted for each video frame in the video data stream.

[0186] In some embodiments, taking the parametric 3D reconstruction of the human body using the SMPL model as an example, multiple skeletal key points and multiple facial key points of the SMPL model are determined in step 303. Then, in step 304, the two-dimensional pose of each skeletal key point can be extracted from the video data (or a video frame), and the two-dimensional pose of each skeletal key point of the SMPL model is used as the skeletal pose information of the entity object. Similarly, the two-dimensional pose of each facial key point can be extracted from the video data (or a video frame), and the two-dimensional pose of each facial key point of the SMPL model is used as the facial pose information of the entity object.

[0187] In steps 303-304 above, a possible implementation method for extracting posture information of the entity object based on the video data is provided. This posture information represents the body posture and facial expression of the entity object in the video data. Therefore, this posture information refers to the two-dimensional position of the entity object in the video data. Here, the decomposition of posture information into skeletal posture information and facial posture information is used as an example for illustration.

[0188] In other embodiments, if the animation generation focuses more on facial expressions (for example, a USB external camera focuses only on the head and neck), then limb movements may not be modeled, and in this case, there is no need to extract skeletal posture information. Alternatively, if the animation generation focuses more on limb movements (such as performing some limb performances), then facial expressions may not be modeled, and in this case, there is no need to extract facial posture information. The embodiments of the present application do not specifically limit this.

[0189] In yet other embodiments, steps 303-304 above provide an implementation method for 3D reconstruction of the human body based on a parametric approach. This allows for the description of highly complex human meshes by extracting only a small number of parameters. For example, SMPL requires only 72+10 parameters to describe a human mesh with 6890 vertices, resulting in high 3D reconstruction efficiency. Of course, in addition to parametric methods, non-parametric methods can also be used for 3D reconstruction of the human body, in which case a high-dimensional human mesh is directly reconstructed. The present embodiments do not specifically limit the 3D reconstruction method.

[0190] The posture information includes skeleton posture information and facial posture information.

[0191] 305. The terminal reconstructs the motion data of the skeleton key points of the virtual image based on the skeleton posture information.

[0192] Among them, the motion data includes the three-dimensional posture of the skeleton key points. In other words, the motion data represents the three-dimensional limb posture of the virtual image obtained after three-dimensional reconstruction of the two-dimensional limb posture of the physical object. Therefore, the motion data refers to the three-dimensional skeletal posture of the virtual image simulated based on the physical object.

[0193] As an example, the above three-dimensional pose indicates position information and attitude information. Then, for any skeletal key point, six pose parameters can be used to characterize its three-dimensional pose, of which three pose parameters characterize the position coordinates (i.e., position information) in the x, y, and z three-dimensional spatial coordinate system, and the other three pose parameters characterize the rotation angle (i.e., attitude information) in the x, y, and z three-dimensional spatial coordinate system. As an example, the above three rotation angles are collectively referred to as Euler angles, which include: pitch angle Pitch, yaw angle Yaw, and roll angle Roll. Pitch angle Pitch represents the angle of rotation around the x-axis, yaw angle Yaw represents the angle of rotation around the y-axis, and roll angle Roll represents the angle of rotation around the z-axis.

[0194] In some embodiments, the terminal performs three-dimensional skeleton reconstruction on the physical object based on the skeleton posture information extracted in step 304 to obtain motion data of the virtual image. Since the skeleton posture information is provided as a two-dimensional pose of multiple skeleton key points, and the motion data is provided as a three-dimensional pose of the multiple skeleton key points, for each skeleton key point, three-dimensional skeleton reconstruction refers to the process of reconstructing its three-dimensional pose based on the two-dimensional pose of the skeleton key point.

[0195] In some embodiments, a pose mapping relationship is established from a two-dimensional plane coordinate system to a three-dimensional space coordinate system. For each skeletal key point, the two-dimensional pose of the skeletal key point is mapped based on the pose mapping relationship to obtain the three-dimensional pose of the skeletal key point. Then, the three-dimensional pose of each skeletal key point in step 303 is used as the motion data of the virtual image. In some embodiments, the pose mapping relationship is established by a three-dimensional reconstruction algorithm. The three-dimensional reconstruction algorithm can be implemented by a computer program or a machine learning model. The three-dimensional reconstruction algorithm can be integrated into a game engine such as UE4 in the form of a plug-in, which is not specifically limited in the embodiments of the present application.

[0196] It should be noted that when the video data stream is obtained in step 301, the skeletal posture information of the physical object will be extracted frame by frame in step 304, so that frame-by-frame three-dimensional skeleton reconstruction can be achieved in step 305 to obtain the action data of the virtual image in each animation frame.

[0197] 306. The terminal reconstructs the expression data of the facial key points of the virtual image based on the facial posture information.

[0198] Among them, the expression data includes the three-dimensional posture of the facial key points. In other words, the expression data represents the three-dimensional expression posture of the virtual image obtained after three-dimensional reconstruction of the two-dimensional expression posture of the physical object. Therefore, the expression data refers to the three-dimensional facial posture of the virtual image simulated based on the physical object.

[0199] As an example, the above three-dimensional pose indicates position information and posture information. Then, for any facial key point, six pose parameters can be used to represent its three-dimensional pose. Among them, three pose parameters represent the position coordinates (i.e., position information) in the x, y, and z three-dimensional spatial coordinate system, and the other three pose parameters represent the rotation angles (i.e., posture information) in the x, y, and z three-dimensional spatial coordinate system. As an example, the above three rotation angles are collectively referred to as Euler angles. Euler angles include: pitch angle Pitch, yaw angle Yaw, and roll angle Roll. Pitch angle Pitch represents the angle of rotation around the x-axis, yaw angle Yaw represents the angle of rotation around the y-axis, and roll angle Roll represents the angle of rotation around the z-axis. For facial key points, the pitch angle Pitch can be regarded as the angle of "nodding", the yaw angle Yaw can be regarded as the angle of "shaking the head", and the roll angle Roll can be regarded as the angle of "tilting / shaking the head".

[0200] In some embodiments, the terminal performs three-dimensional facial reconstruction on the physical object based on the facial pose information extracted in step 304 to obtain expression data for the avatar. Since the facial pose information is provided as a two-dimensional pose of multiple facial key points, and the expression data is provided as a three-dimensional pose of the multiple facial key points, three-dimensional facial reconstruction refers to the process of reconstructing the three-dimensional pose of each facial key point based on the two-dimensional pose of the facial key point.

[0201] In some embodiments, a pose mapping relationship is established from a two-dimensional plane coordinate system to a three-dimensional space coordinate system. For each facial key point, the two-dimensional pose of the facial key point is mapped based on the pose mapping relationship to obtain the three-dimensional pose of the facial key point. The three-dimensional pose of each facial key point in step 303 is then used as the expression data of the avatar. In some embodiments, the pose mapping relationship is established using a three-dimensional reconstruction algorithm. The three-dimensional reconstruction algorithm can be implemented as a computer program or a machine learning model. The three-dimensional reconstruction algorithm can be integrated into a game engine such as UE4 in the form of a plug-in, which is not specifically limited in this embodiment of the present application.

[0202] It should be noted that when the video data stream is obtained in step 301, the facial posture information of the entity object will be extracted frame by frame in step 304, so as to realize frame-by-frame three-dimensional facial reconstruction in step 306 and obtain the expression data of the virtual image in each animation frame.

[0203] In steps 305-306 above, a possible implementation method is provided for performing a three-dimensional reconstruction of the physical object based on the posture information of the physical object to obtain motion data and expression data of the virtual image, wherein the motion data represents the physical movements of the virtual image reconstructed based on the physical posture, and the expression data represents the facial expressions of the virtual image reconstructed based on the expression posture. Because the posture information is decomposed into skeletal posture information and facial posture information, the motion data can be reconstructed based on the skeletal posture information, and the expression data can be reconstructed based on the facial posture information, thereby accurately reconstructing the motion data and expression data, respectively, in 3D, thereby improving the accuracy and rendering effect of animation generation.

[0204] In other embodiments, if the animation generation focuses more on facial expressions (for example, a USB external camera focuses only on the head and neck), then limb movements may not be modeled. In this case, there is no need to extract skeletal posture information or perform 3D skeletal reconstruction. It is only necessary to configure the action data of the virtual image to a standing action, or to configure the changes of its animation nodes according to the animation state machine; or, if the animation generation focuses more on limb movements (such as performing some limb performances), then facial expressions may not be modeled. In this case, there is no need to extract facial posture information or perform 3D facial reconstruction. It is only necessary to configure the expression data of the virtual image to be expressionless, or to drive the lip shape changes according to the audio content. The embodiments of the present application do not specifically limit this.

[0205] 307. The terminal determines the skin weight of each skeletal component of the avatar relative to each vertex in the skeletal skin of the avatar.

[0206] As an example, the skin weight represents the extent to which the vertex is influenced by the bone component.

[0207] In some embodiments, the terminal first obtains the mesh data of the virtual image, and then binds the mesh data to each skeletal component to obtain the skeletal skin of the virtual image. The skeletal skin includes multiple vertices. For each vertex, the skin weight of each skeletal component relative to the vertex is determined.

[0208] In some embodiments, as shown in Figure 6, Figure 6 is a schematic diagram of a skeleton skinning method provided in an embodiment of the present application. The original model (i.e., three-dimensional model) and the skeleton component of the virtual image are obtained, and the original model and the skeleton component are bound to obtain the skeleton skin of the virtual image. The skeleton skin here refers to the virtual image Mesh bound with the skeleton component. The skeleton skin can actually be represented as a set of Mesh vertex sets. For each vertex in the Mesh vertex set, the skin weight of each skeleton component relative to the vertex needs to be calculated. For example, in the case of N (N≥2) skeleton components, for vertex v, the skin weight of skeleton component 1 for vertex v is calculated, and the skin weight of skeleton component 2 for vertex v is calculated, and so on, until the skin weight of skeleton component N for vertex v is calculated. In this way, vertex v will get N skin weights, and each skin weight represents the degree to which vertex v is affected by the corresponding skeleton component. In this way, the N skin weights can reflect which skeleton components affect vertex v and the weight of the influence.

[0209] Below, a possible method for obtaining skeleton skinning will be described through steps A1 to A2.

[0210] A1. The terminal exports mesh data of the virtual image based on the three-dimensional model of the virtual image.

[0211] As an example, the three-dimensional model performs default body movements and has default facial expressions. For example, the default body movements refer to body movements that do not perform specific meanings, such as static standing movements with limbs placed naturally. For example, the default facial expressions refer to facial expressions that do not have specific emotions, such as static expressionless faces with facial features in natural forms. The mesh data represents the meshed outer surface of the three-dimensional model.

[0212] In some embodiments, the terminal stores three-dimensional models of various types of virtual images in the form of point clouds. Therefore, based on the image ID of the virtual image, the three-dimensional model of the virtual image stored in the form of point clouds can be queried, the three-dimensional model in the form of point clouds can be baked, and the mesh data of the three-dimensional model (i.e., the original mesh) can be exported.

[0213] In other embodiments, if the virtual image supports the user to perform personalized face-pinching operations, that is, the user can edit the facial configuration parameters of the virtual image (such as eye distance, pupil distance, jaw width, philtrum length, etc.) through the face-pinching operation, then the terminal can also store the user's face-pinching parameters, and first adjust the point cloud position in the three-dimensional model of the virtual image according to the face-pinching parameters, and then export the mesh data based on the adjusted three-dimensional model, thereby ensuring that the exported mesh data conforms to the user-defined face-pinching parameters.

[0214] A2. The terminal binds the mesh data of the parts associated with each skeletal component of the three-dimensional model to obtain the skeletal skin of the virtual image.

[0215] As an example, the skeletal component represents the skeleton of the part.

[0216] In some embodiments, since the three-dimensional model includes a 3D skeletal model and a 3D facial model, the 3D skeletal model is split into multiple skeletal components, and the 3D facial model is treated as a special skeletal component that controls facial expressions. In this way, the avatar's body movements and facial expressions can be controlled by all skeletal components. Therefore, the terminal can bind each skeletal component of the avatar to the corresponding part in the mesh data, thereby achieving skeletal skinning of the three-dimensional model.

[0217] In steps A1-A2, since different avatars may have different body shapes, it is only necessary to replace the skeleton components to achieve the export of skeleton skins for different avatars, thereby being able to expand to different types of avatars and having wide versatility. In other embodiments, the terminal may also pre-store the skeleton skin of each avatar, so that it is only necessary to query the skeleton skin of the avatar based on the image ID of the avatar. Furthermore, assuming that the avatar is configured with N skeleton components, after obtaining the skeleton skin of the avatar, N skin weights for each vertex in the skeleton skin are exported.

[0218] 308. The terminal determines the posture reconstruction data of each skeletal component based on the motion data and the expression data.

[0219] In some embodiments, based on the action data in step 305, the various skeletal components contained in the 3D skeletal model can be controlled to present certain body movements. Similarly, based on the expression data in step 306, the specific skeletal components contained in the 3D facial model can be controlled to present certain facial expressions. By combining the body movements and facial expressions of the virtual image, the animation data of the virtual image can be synthesized.

[0220] In some embodiments, in order to facilitate the synthesis of animation data, the pose reconstruction data of each skeletal component is determined in units of skeletal components, so that it is convenient to apply deformation (or offset) to each vertex in the skeletal skin according to the skin weight.

[0221] Below, through steps B1 to B4, a possible method for obtaining pose reconstruction data will be provided. By obtaining pose reconstruction data based on skeletal components, the pose reconstruction data of skeletal components can be directly used to implement skeletal animation to improve the efficiency of animation generation. It can also be further synthesized into vertex-by-vertex animation with vertex precision, thereby improving the precision of animation generation and enhancing the flexibility of animation generation.

[0222] B1. The terminal determines the reconstruction key points contained in each skeletal component of the virtual image.

[0223] In some embodiments, for each of the N (N≥2) skeletal components of the virtual image, multiple reconstruction key points contained in the skeletal component can be determined from the three-dimensional model of the virtual image, wherein the reconstruction key points include at least one of skeletal key points or facial key points. This embodiment of the present application does not specifically limit this.

[0224] B2. When the reconstructed key point includes a skeleton key point, the terminal determines the three-dimensional pose of the reconstructed key point based on the motion data.

[0225] In some embodiments, if the multiple reconstructed key points in step B1 include at least one skeletal key point, then based on the motion data in step 305, since the motion data includes the three-dimensional pose of each skeletal key point, querying the motion data can obtain the three-dimensional pose of the at least one skeletal key point, thereby finding the three-dimensional pose of each skeletal key point included in the reconstructed key points. Of course, if the multiple reconstructed key points in step B1 do not include any skeletal key points, there is no need to perform this step B2.

[0226] B3. When the reconstructed key point includes a facial key point, the terminal determines the three-dimensional pose of the reconstructed key point based on the expression data.

[0227] In some embodiments, if the multiple reconstructed key points in step B1 include at least one facial key point, then based on the expression data in step 306, since the expression data includes the three-dimensional pose of each facial key point, querying the expression data can obtain the three-dimensional pose of the at least one facial key point, thereby finding the three-dimensional pose of each facial key point included in the reconstructed key points. Of course, if the multiple reconstructed key points in step B1 do not include any facial key points, step B3 does not need to be performed.

[0228] B4. The terminal determines the three-dimensional pose of each reconstruction key point contained in the skeleton component as the pose reconstruction data of the skeleton component.

[0229] In some embodiments, the three-dimensional pose of each reconstructed key point obtained in steps B2 to B3 is determined as the pose reconstruction data of the current skeletal component. Since the three-dimensional pose of each reconstructed key point involves two dimensions: position information and rotation information, for example, 6 pose parameters are used to represent the three-dimensional pose of each reconstructed key point. Then 3 of the pose parameters represent the position coordinates (i.e., position information) in the x, y, z three-dimensional space coordinate system, and the other 3 pose parameters represent the rotation angle (i.e., rotation information) in the x, y, z three-dimensional space coordinate system. In this case, the pose reconstruction data of each skeletal component can be regarded as including displacement reconstruction information and rotation reconstruction information.

[0230] In steps B1 to B4, a possible method for acquiring pose reconstruction data is provided. This allows for acquiring pose reconstruction data for skeletal components, regardless of whether they involve facial key points or skeletal key points, ensuring the accuracy of the pose reconstruction data. Furthermore, upon acquiring the pose reconstruction data for each skeletal component, skeletal animation can be directly implemented using the pose reconstruction data, improving animation generation efficiency. Alternatively, step 309 can be executed to further synthesize vertex-by-vertex animation with high vertex precision, thereby enhancing the sophistication and flexibility of animation generation.

[0231] 309. The terminal determines the vertex position of each vertex based on the posture reconstruction data and the skin weight.

[0232] In some embodiments, after the skeleton is skinned, the skeleton components are affected by the motion data or expression data, and when displacement or rotation occurs (i.e., posture changes), it will naturally affect the vertex position of the vertices in the bound mesh data. Therefore, the vertex position of each vertex in the mesh data needs to be recalculated. This is equivalent to applying deformation to the set vertices on the mesh data through the motion data and expression data, thereby controlling the position offset of the vertices in the skeleton skin, so that the body movements and expression postures presented by the virtual image can be controlled from the vertex level, and are not limited to certain pre-set fixed animation behaviors. Therefore, the reconstruction accuracy of body movements and facial expressions is higher, and the degree of restoration and realism are higher.

[0233] The following will introduce the method of determining the vertex position of a single vertex through steps C1 to C2, taking the position transformation process of a single vertex in skeleton skinning as an example.

[0234] C1. For each vertex of the skeleton skin, the terminal determines the associated skeleton component of the vertex from the skeleton components based on the skin weights of the skeleton components relative to the vertex.

[0235] In some embodiments, for each vertex in the skeleton skin obtained in step 307, N (N≥2) skeleton components are calculated relative to the N skin weights of the vertex, so that at least one associated skeleton component of the vertex can be found from the N skeleton components based on the N skin weights of the vertex.

[0236] In some embodiments, an influence threshold can be pre-configured. If the skin weight of any skeletal component is greater than the influence threshold, the skeletal component is determined to be an associated skeletal component of the vertex. The influence threshold is a pre-configured value or a default value. For example, the influence threshold is 0, or a value greater than or equal to 0 such as 0.2 or 0.5. The value of the influence threshold is not specifically limited here. Furthermore, if the skin weights of all skeletal components are less than or equal to the influence threshold, then there is no need to recalculate the vertex position for the vertex, skip the position calculation of the current vertex, and start calculating the vertex position for the next vertex.

[0237] In other embodiments, the N skeletal components can also be sorted in descending order of skinning weight, and the skeletal components in the top K positions can be selected as the K associated skeletal components of the vertex, where K≥1. This means that there is no need to configure an influence threshold, but rather the topK associated skeletal components are preferred to recalculate the vertex position. The embodiments of the present application do not specifically limit the method for selecting associated skeletal components.

[0238] C2. The terminal determines the vertex position of the vertex based on the pose reconstruction data of the associated skeletal component and the skin weight.

[0239] In some embodiments, for each vertex in the skeletal skin, step C1 can filter out K (K ≥ 1) associated skeletal components from the N skeletal components. Thus, the vertex position of the vertex can be recalculated using only the K pose reconstruction data obtained by the K associated skeletal components in step 308, as well as the K skin weights assigned to the vertex by each of the K associated skeletal components. In some embodiments, the K pose reconstruction data are weighted and summed using the K skin weights to obtain an offset vector for the vertex. This offset vector is then applied to the vertex's initial position in the skeletal skin to obtain the updated vertex position.

[0240] As an example, as shown in Figure 7, Figure 7 is a schematic diagram of an animation drive process provided by an embodiment of the present application. The posture reconstruction data of the N skeletal components obtained in step 308 are regarded as a skeletal motion data set, so the skeletal motion data set contains the posture reconstruction data of the N skeletal components, and each posture reconstruction data contains a displacement reconstruction information and a rotation reconstruction information. The above skeletal motion data set is called driving data. After the driving data is input into the virtual image animation system, the virtual image animation system can recalculate the vertex position of each vertex in the skeleton skin, and then update the vertex position of each vertex in the skeleton skin, and then input it into step 310 for rendering, which is equivalent to calculating whether each vertex will be deformed (or offset) under the influence of the driving data, and the vertex position after deformation, so that the animation behavior of the virtual image can be controlled from the vertex granularity.

[0241] In steps C1 to C2, for each vertex, only the skin weights and pose reconstruction data of K associated skeletal components need to be considered. This reduces the computational complexity and complexity of vertex calculation and improves the efficiency of animation generation. In other embodiments, the associated skeletal components may not be filtered, and the skin weights and pose reconstruction data of N skeletal components may be directly used for each vertex to recalculate the vertex position. This can fully consider the impact of all skeletal components on each vertex, thereby improving the precision of animation reconstruction.

[0242] 310. The terminal synthesizes animation data of the virtual image based on the appearance resource of the virtual image and the vertex position.

[0243] Among them, appearance resources are used to control the external performance of the virtual image. For example, the appearance resources include but are not limited to: hair, skin, eyes, clothing, accessories, special effects, etc.

[0244] The animation data represents that the virtual image wears the appearance resource, presents the facial expression, and performs the body movement.

[0245] In some embodiments, based on the avatar's ID, the avatar's appearance resources are retrieved, the retrieved appearance resources are combined with the skeleton skin to synthesize animation data for the avatar, and the animation data is rendered and visualized so that the avatar wears its own appearance resources and exhibits body movements and facial expressions that simulate a physical object. The positions of each vertex in the skeleton skin are recalculated using the method provided in step 309.

[0246] In some embodiments, the GPU's rendering pipeline is driven based on the recalculated vertex skin and the avatar's own appearance resources. The skeletal skin is then subjected to a series of rendering processes, including vertex shading, rasterization, and pixel shading. Ultimately, the avatar's animation data is visualized, i.e., the avatar's animation data is drawn onto the terminal's display screen, thereby displaying the avatar's animation data. During the real-time animation generation process, the avatar's animation can be played by simply displaying each animation frame one by one, achieving real-time, dynamic animation drive.

[0247] In some embodiments, a rendering hardware interface (RHI) thread, namely RHIThread, is created in the terminal, and the appearance resources of the above-mentioned virtual image and the deformed skeleton skin are submitted to the RHIThread. The RHIThread then executes the drawing commands in the graphics API, drives the rendering pipeline on the GPU to render the virtual image, and obtains an animation frame of the virtual image.

[0248] Below, the GPU rendering process of any animation frame will be explained. The GPU rendering pipeline involves vertex shader, rasterizer and pixel shader. Among them, the vertex shader is a rendering pipeline used to calculate the vertices of the mesh, and the rasterizer is a rendering pipeline used to assemble the results of the vertex shader output into a triangle mesh and rasterize it into discrete pixels according to the configuration. The pixel shader is a rendering pipeline used to perform shading calculations on each discrete pixel after rasterization and complete pixel shading.

[0249] For each vertex in the mesh data, the action and expression data control the vertex position. Of course, posture information such as rotation angle may also affect the vertex's depth value, while the appearance resource affects the vertex's color and depth values. First, the frame buffer's color and depth are cleared. Next, the vertex shader writes the depth of each vertex in the skinned skeleton to the z-buffer. This involves a depth sorting process, which affects transparency and handles occlusion or translucency. Next, the rasterizer performs rasterization. Next, the pixel shader writes the color of each discretized pixel to the color buffer. This is the pixel value of each pixel. Each pixel value is the integration of the color values ​​of all vertices located at that pixel. This color integration process typically takes into account vertex transparency, ultimately resulting in the output of a single animation frame of the avatar on the terminal's display. It's also important to note that the vertex shading stage often involves lighting calculations, with the specific lighting model used being determined by the avatar or rendering engine.

[0250] Among them, the frame buffer is used to store the data of all pixels in the current animation frame. The frame buffer contains a z buffer and a color buffer. The z buffer refers to the depth buffer in the frame buffer, which is used to store the depth value of each pixel in the current animation frame, and the color buffer is used to store the color value of each pixel in the current animation frame.

[0251] In steps 307 to 310, a possible implementation method for synthesizing the animation data of the virtual image based on the appearance resources, the action data and the expression data of the virtual image is provided, which can simulate the animation data of the virtual image based on the video data of the physical object and meet the appearance resources required to be worn by the virtual image itself.

[0252] It should be noted that when the video data stream is obtained in step 301, the skeleton posture information and facial posture information of the entity object will be extracted frame by frame in step 304, and then frame-by-frame three-dimensional skeleton reconstruction will be implemented in step 305 to obtain the action data of the virtual image in each animation frame. Similarly, frame-by-frame three-dimensional facial reconstruction will be implemented in step 306 to obtain the expression data of the virtual image in each animation frame. The vertex position of each vertex in the skeleton skin will be recalculated frame by frame in step 309. Further, in this step 310, the animation data of the virtual image in each animation frame can be simulated frame by frame. Therefore, when the video data includes multiple video frames, the animation data includes multiple animation frames, wherein each animation frame is associated with a video frame, and the body movements and facial expressions of the virtual image in the animation frame match the body movements and facial expressions of the entity object in the video frame. Through the embodiment of the present application, the body movements and facial expressions of the virtual image and the entity object in each frame can be kept consistent, thereby optimizing the rendering effect of the virtual image.

[0253] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0254] The method provided in the embodiment of the present application extracts the body posture and facial expression of the physical object in a plane coordinate system through the video data of the physical object, and then performs three-dimensional reconstruction of the physical object to obtain the body movements and facial expressions of the virtual image in the spatial coordinate system, and then combines the appearance resources of the virtual image to synthesize the animation data of the virtual image, so that the virtual image can simulate and emulate the body movements and facial expressions of the physical object under the drive of the video data. This animation generation process does not rely on expensive motion capture equipment, can be extended to general scenarios such as live broadcasts and games, and can reconstruct animations in real time to meet higher real-time requirements. The animation generation has low latency and high efficiency.

[0255] Furthermore, since animation rendering does not rely on skeletal animation or UE4's native digital human system, the synthesized virtual image animation is not limited to a few pre-set action categories, and there will be no problems such as mechanical feeling or rigid connection during animation presentation. The animation control of the virtual image can be achieved vertex by vertex, and the reconstruction accuracy of its body movements and facial expressions is detailed to the vertex level in the mesh data of the virtual image. It can control the virtual image to accurately imitate the body movements and facial expressions of the physical object with high precision, thereby improving the animation generation accuracy of the virtual image, having high flexibility and controllability, and optimizing the animation effect of the virtual image.

[0256] Next, we will take the video data stream captured by a USB external camera as an example to introduce how to drive the real-time animation generation of virtual images based on the UE4 engine.

[0257] FIG8 is a principle flow chart of an animation generation solution provided by an embodiment of the present application. As shown in FIG8 , a USB external camera is connected to each platform terminal, and each platform terminal and the USB external camera are driven and adapted at the platform layer to complete the communication connection between the USB external camera and the terminal; then, UE4 can call the USB external camera in the form of a plug-in to obtain the collected video data stream; then, UE4 performs data preprocessing (i.e., video format conversion) on the video data stream to obtain a video data stream in a preset video format (referred to as algorithm protocol data for short), wherein the video format conversion can use asynchronous thread acceleration or GPU hardware acceleration; then, the 3D reconstruction algorithm is integrated into UE4 in the form of a plug-in, and the algorithm The protocol data is input into the three-dimensional reconstruction algorithm to obtain the pose reconstruction data of each skeletal component, where each pose reconstruction data includes a displacement reconstruction information and a rotation reconstruction information. The position reconstruction data of each skeletal component are collectively referred to as the algorithm result; then, the algorithm result is input into the animation blueprint of the UE4 virtual image to control the pose change of the skeletal component of the virtual image, thereby affecting the vertex position of the vertices in the skeleton skin. Each recalculated vertex position will affect the animation data of the virtual image, thereby rendering the animation data of the virtual image and visually presenting the appearance resources, body movements and facial expressions of the virtual image, that is, displaying the reconstruction results of the human skeleton and facial expressions of the virtual image.

[0258] Figure 9 is a logical principle diagram of an animation generation solution provided by an embodiment of the present application. As shown in Figure 9, adaptation of various platforms is implemented at the driver layer, so that the terminal can detect the connection status of the USB external camera and set the shooting parameters of the USB external camera, such as the acquisition frame rate, resolution, etc., so that the terminal can smoothly drive the USB external camera to shoot the video data stream in real time; further, at the algorithm layer, the video data stream collected by the USB external camera is preprocessed and converted into a video data stream in a preset video format adapted to the three-dimensional reconstruction algorithm, and then the posture information of the physical object is extracted from the video data stream in the preset video format, and then the motion data and expression data of the virtual image are three-dimensionally reconstructed; further, at the rendering layer, the animation data of the virtual image is driven to be synthesized based on the motion data and expression data of the virtual image, combined with the appearance resources of the virtual image, and the synthesized animation data is rendered, so that the body movements and facial expressions of the virtual image in the final rendering result highly simulate the body movements and facial expressions made by the physical object, thereby meeting the requirements of high-quality animation rendering of the virtual image in any scene with high real-time requirements.

[0259] In the above animation generation solution, video data streams are captured using a USB external camera. Due to their high compatibility and ease of adaptation to terminals on any platform, USB external cameras can be used on a wide range of terminals, including mobile phones and personal computers. They can also be used on display devices with built-in cameras, such as conference room monitors, projectors, and shopping mall displays. This approach covers a wider range of applicable scenarios, making real-time animation rendering of virtual avatars, such as digital humans, no longer limited to professional film and television production settings. Furthermore, this solution streamlines the processing chain within UE4, from capturing video data using a USB external camera, to algorithmically reconstructing motion and expression data, and driving virtual avatars to simulate real-life behavior. This expands the application scenarios of avatars and enables high-quality rendering and driving of avatars for a wide range of display devices. As long as the terminal supports UE4, this solution can be used out-of-the-box, with low cost and cross-platform support. This significantly improves animation generation efficiency, reduces the technical threshold and synthesis costs, and improves the user experience.

[0260] Figure 10 is a structural schematic diagram of a virtual image animation generation device provided by an embodiment of the present application. As shown in Figure 10, the device includes: an acquisition module 1001, configured to acquire video data of an entity object; an extraction module 1002, configured to extract posture information of the entity object based on the video data, and the posture information represents the body posture and facial expression posture of the entity object presented in the video data; a reconstruction module 1003, configured to perform three-dimensional reconstruction of the entity object based on the posture information to obtain action data and facial expression data of the virtual image, the action data represents the body movement of the virtual image obtained based on the body posture reconstruction, and the facial expression data represents the facial expression of the virtual image obtained based on the facial expression posture reconstruction; a synthesis module 1004, configured to synthesize animation data of the virtual image based on the appearance resource of the virtual image, the action data and the facial expression data, and the animation data represents that the virtual image wears the appearance resource, presents the facial expression and performs the body movement.

[0261] The device provided in the embodiment of the present application extracts the body posture and facial expression of the physical object in a plane coordinate system through the video data of the physical object, and then performs three-dimensional reconstruction of the physical object to obtain the body movements and facial expressions of the virtual image in the spatial coordinate system, and then combines the appearance resources of the virtual image to synthesize the animation data of the virtual image, so that the virtual image can simulate and emulate the body movements and facial expressions of the physical object under the drive of the video data. This animation generation process does not rely on expensive motion capture equipment, can be extended to general scenarios such as live broadcasts and games, and can reconstruct animations in real time to meet higher real-time requirements. The animation generation has low latency and high efficiency.

[0262] In some embodiments, the extraction module 1002 is configured to: determine the skeletal key points and facial key points of the entity object; based on the video data, extract the skeletal posture information of the skeletal key points and the facial posture information of the facial key points, the skeletal posture information represents the two-dimensional posture of the skeletal key points, and the facial posture information represents the two-dimensional posture of the facial key points; and combine the skeletal posture information and the facial posture information to form the posture information of the entity object.

[0263] In some embodiments, the posture information includes skeletal posture information and facial posture information, and the reconstruction module 1003 is configured to: reconstruct the motion data of the skeletal key points of the virtual image based on the skeletal posture information, and the motion data includes the three-dimensional posture of the skeletal key points; reconstruct the expression data of the facial key points of the virtual image based on the facial posture information, and the expression data includes the three-dimensional posture of the facial key points.

[0264] In some embodiments, based on the device composition of Figure 10, the synthesis module 1004 includes: a weight determination unit, configured to determine the skin weight of each skeletal component of the virtual image relative to each vertex in the skeletal skin of the virtual image, and the skin weight represents the degree to which the vertex is affected by the skeletal component; a posture determination unit, configured to determine the posture reconstruction data of each skeletal component based on the action data and the expression data; a position determination unit, configured to determine the vertex position of each vertex based on the posture reconstruction data and the skin weight; an animation synthesis unit, configured to synthesize the animation data based on the appearance resource and the vertex position.

[0265] In some embodiments, based on the device composition of Figure 10, the device also includes: an export module, configured to export the mesh data of the virtual image based on the three-dimensional model of the virtual image; a binding module, configured to bind the mesh data of the part associated with each skeletal component of the three-dimensional model to obtain the skeletal skin of the virtual image.

[0266] In some embodiments, the posture determination unit is configured to: determine the reconstruction key points contained in each skeletal component of the virtual image; when the reconstruction key points include skeletal key points, determine the three-dimensional posture of the reconstruction key points based on the motion data; when the reconstruction key points include facial key points, determine the three-dimensional posture of the reconstruction key points based on the expression data; and determine the three-dimensional posture of each reconstruction key point contained in the skeletal component as the posture reconstruction data of the skeletal component.

[0267] In some embodiments, the position determination unit is configured to: for each vertex of the skeleton skin, determine the associated skeleton component of the vertex from the various skeleton components based on the skin weights of the various skeleton components relative to the vertex; and determine the vertex position of the vertex based on the pose reconstruction data of the associated skeleton component and the skin weight.

[0268] In some embodiments, based on the device composition of Figure 10, the acquisition module 1001 includes: a shooting unit, configured to shoot video data of the physical object based on an external camera; a conversion unit, configured to convert the video data from a video format supported by the external camera to a preset video format, which is a format that supports three-dimensional reconstruction of the physical object.

[0269] In some embodiments, the conversion unit is configured to perform at least one of the following: starting a sub-thread configured for format conversion, converting the video data from the video format supported by the external camera to the preset video format through the sub-thread; calling a drawing command of a graphics processor GPU, converting the video data from the video format supported by the external camera to the preset video format through the GPU.

[0270] In some embodiments, when the video data includes multiple video frames, the animation data includes multiple animation frames, wherein each animation frame is associated with a video frame, and the body movements and facial expressions of the virtual image in the animation frame match the body movements and facial expressions of the physical object in the video frame.

[0271] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0272] It should be noted that the aforementioned embodiments of the avatar animation generation device, when generating avatar animation data, only illustrate the division of the aforementioned functional modules. In actual applications, the aforementioned functions can be assigned to different functional modules as needed, that is, the internal structure of the electronic device can be divided into different functional modules to complete all or part of the functions described above. In addition, the avatar animation generation device and the avatar animation generation method embodiment provided in the aforementioned embodiments are based on the same concept. The specific implementation process is detailed in the avatar animation generation method embodiment and will not be repeated here.

[0273] FIG11 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. As shown in FIG11 , the electronic device is taken as a terminal for example. Generally, the terminal 1100 includes: a processor 1101 and a memory 1102 .

[0274] In some embodiments, processor 1101 includes one or more processing cores, such as a quad-core processor or an octa-core processor. In some embodiments, processor 1101 is implemented in hardware using at least one of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). In some embodiments, processor 1101 includes a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, processor 1101 is integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the display screen. In some embodiments, processor 1101 also includes an AI (Artificial Intelligence) processor, which is used to handle computing operations related to machine learning.

[0275] In some embodiments, the memory 1102 includes one or more computer-readable storage media, which in some embodiments are non-transitory. In some embodiments, the memory 1102 also includes high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1102 is used to store at least one program code, which is executed by the processor 1101 to implement the virtual image animation generation method provided in various embodiments of the present application.

[0276] In some embodiments, terminal 1100 may optionally include a peripheral device interface 1103 and at least one peripheral device. Processor 1101, memory 1102, and peripheral device interface 1103 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 1103 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 1104, a display screen 1105, a camera assembly 1106, an audio circuit 1107, and a power supply 1108.

[0277] The peripheral device interface 1103 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 1101 and the memory 1102. In some embodiments, the processor 1101, the memory 1102, and the peripheral device interface 1103 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1101, the memory 1102, and the peripheral device interface 1103 are implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0278] The RF circuit 1104 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1104 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1104 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. In some embodiments, the RF circuit 1104 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. In some embodiments, the RF circuit 1104 communicates with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, metropolitan area networks, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1104 also includes circuits related to NFC (Near Field Communication), which is not limited in this application.

[0279] The display screen 1105 is used to display a UI (User Interface). In some embodiments, the UI includes graphics, text, icons, videos, and any combination thereof. When the display screen 1105 is a touch screen display, the display screen 1105 also has the ability to collect touch signals on the surface or above the surface of the display screen 1105. The touch signal can be input as a control signal to the processor 1101 for processing. In some embodiments, the display screen 1105 is also used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, there is one display screen 1105, which is set on the front panel of the terminal 1100; in other embodiments, there are at least two display screens 1105, which are respectively set on different surfaces of the terminal 1100 or in a folding design; in some embodiments, the display screen 1105 is a flexible display screen, which is set on the curved surface or folding surface of the terminal 1100. In some embodiments, the display screen 1105 is set into a non-rectangular irregular shape, that is, a special-shaped screen. In some embodiments, the display screen 1105 is made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0280] The camera assembly 1106 is used to capture images or videos. In some embodiments, the camera assembly 1106 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1106 also includes a flash. In some embodiments, the flash is a monochrome temperature flash, or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which is used for light compensation at different color temperatures.

[0281] In some embodiments, the audio circuit 1107 includes a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1101 for processing, or input them into the radio frequency circuit 1104 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there are multiple microphones, which are respectively arranged at different parts of the terminal 1100. In some embodiments, the microphone is an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1101 or the radio frequency circuit 1104 into sound waves. In some embodiments, the speaker is a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1107 also includes a headphone jack.

[0282] Power supply 1108 is used to power various components in terminal 1100. In some embodiments, power supply 1108 is AC power, DC power, a disposable battery, or a rechargeable battery. When power supply 1108 includes a rechargeable battery, the rechargeable battery supports wired charging or wireless charging. The rechargeable battery is also configured to support fast charging technology.

[0283] In some embodiments, the terminal 1100 further includes one or more sensors 1110 , including but not limited to: an acceleration sensor 1111 , a gyroscope sensor 1112 , a pressure sensor 1113 , an optical sensor 1114 , and a proximity sensor 1115 .

[0284] In some embodiments, the accelerometer 1111 detects the magnitude of acceleration along the three coordinate axes of the coordinate system established by the terminal 1100. For example, the accelerometer 1111 is used to detect the components of gravity acceleration along the three coordinate axes. In some embodiments, the processor 1101 controls the display screen 1105 to display the user interface in a landscape or portrait view based on the gravity acceleration signal collected by the accelerometer 1111. The accelerometer 1111 is also used to collect game or user motion data.

[0285] In some embodiments, the gyroscope sensor 1112 detects the orientation and rotation angle of the terminal 1100. The gyroscope sensor 1112 and the acceleration sensor 1111 collaborate to collect the user's 3D movements on the terminal 1100. The processor 1101 implements the following functions based on the data collected by the gyroscope sensor 1112: motion sensing (for example, changing the UI based on the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0286] In some embodiments, the pressure sensor 1113 is disposed on the side frame of the terminal 1100 and / or the lower layer of the display screen 1105. When the pressure sensor 1113 is disposed on the side frame of the terminal 1100, it can detect the user's grip signal of the terminal 1100, and the processor 1101 performs left and right hand recognition or shortcut operations based on the grip signal collected by the pressure sensor 1113. When the pressure sensor 1113 is disposed on the lower layer of the display screen 1105, the processor 1101 controls the operable controls on the UI interface based on the user's pressure operation on the display screen 1105. The operable controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0287] Optical sensor 1114 is used to detect ambient light intensity. In one embodiment, processor 1101 controls the display brightness of display screen 1105 based on the ambient light intensity detected by optical sensor 1114. Specifically, when the ambient light intensity is high, the display brightness of display screen 1105 is increased; when the ambient light intensity is low, the display brightness of display screen 1105 is decreased. In another embodiment, processor 1101 also dynamically adjusts the capture parameters of camera assembly 1106 based on the ambient light intensity detected by optical sensor 1114.

[0288] Proximity sensor 1115, also known as a distance sensor, is typically located on the front panel of terminal 1100. Proximity sensor 1115 is used to detect the distance between the user and the front of terminal 1100. In one embodiment, when proximity sensor 1115 detects that the distance between the user and the front of terminal 1100 is gradually decreasing, processor 1101 controls display screen 1105 to switch from the screen-on state to the screen-off state. When proximity sensor 1115 detects that the distance between the user and the front of terminal 1100 is gradually increasing, processor 1101 controls display screen 1105 to switch from the screen-off state to the screen-on state.

[0289] Those skilled in the art will appreciate that the structure shown in FIG11 does not limit the terminal 1100 , and the terminal 1100 may include more or fewer components than shown, or combine certain components, or adopt a different component arrangement.

[0290] FIG12 is a schematic diagram of the structure of another electronic device provided in an embodiment of the present application. As shown in FIG12 , taking the electronic device as a server as an example, the server 1200 may have relatively large differences due to different configurations or performances. The server 1200 includes one or more processors (Central Processing Units, CPUs) 1201 and one or more memories 1202, wherein the memories 1202 store at least one computer-executable instruction, and the at least one computer-executable instruction is loaded and executed by the one or more processors 1201 to implement the animation generation method of the virtual image provided in each of the above embodiments. In some embodiments, the server 1200 also has components such as a wired or wireless network interface, a keyboard, and an input / output interface for input and output. The server 1200 also includes other components for implementing device functions, which are not described in detail here.

[0291] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including at least one computer-executable instruction. The at least one computer-executable instruction can be executed by a processor in an electronic device to implement the methods for generating an animation of a virtual character in each of the above-described embodiments. For example, the computer-readable storage medium includes ROM (Read-Only Memory), RAM (Random-Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage device.

[0292] In an exemplary embodiment, a computer program product is also provided, comprising one or more computer-executable instructions stored in a computer-readable storage medium. One or more processors of an electronic device can read the one or more computer-executable instructions from the computer-readable storage medium and execute the one or more computer-executable instructions, thereby enabling the electronic device to perform the method for generating an animation of an avatar in the above-described embodiment.

[0293] Those skilled in the art will understand that all or part of the steps to implement the above embodiments can be completed by hardware, or can be completed by instructing related hardware through a program. In some embodiments, the program is stored in a computer-readable storage medium. In some embodiments, the above-mentioned storage medium is a read-only memory, a disk, or an optical disk, etc.

[0294] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for generating an animation of a virtual image, the method being executed by an electronic device, the method comprising: Get the video data of the entity object; Extracting posture information of the entity object based on the video data, wherein the posture information represents a body posture and facial expression posture of the entity object presented in the video data; Based on the posture information, the physical object is three-dimensionally reconstructed to obtain motion data and expression data of the virtual image, wherein the motion data represents the body motion of the virtual image reconstructed based on the body posture, and the expression data represents the facial expression of the virtual image reconstructed based on the expression posture; Based on the appearance resources of the virtual image, the action data and the expression data, animation data of the virtual image is synthesized, and the animation data represents that the virtual image wears the appearance resources, presents the facial expression and performs the body movement.

2. The method according to claim 1, wherein: The step of extracting the posture information of the entity object based on the video data includes: Determine the skeleton key points and the facial key points of the entity object; Based on the video data, extracting the skeleton posture information of the skeleton key points and the facial posture information of the facial key points, wherein the skeleton posture information represents the two-dimensional position and posture of the skeleton key points, and the facial posture information represents the two-dimensional position and posture of the facial key points; The skeleton posture information and the facial posture information are combined into posture information of the entity object.

3. The method according to claim 1 or 2, wherein: The posture information includes skeleton posture information and facial posture information. Based on the posture information, the entity object is three-dimensionally reconstructed to obtain motion data and expression data of the virtual image, including: Based on the skeleton posture information, reconstructing the motion data of the skeleton key points of the virtual image, wherein the motion data includes the three-dimensional position and posture of the skeleton key points; Based on the facial posture information, expression data of facial key points of the virtual image are reconstructed, and the expression data includes three-dimensional positions and postures of the facial key points.

4. The method according to any one of claims 1 to 3, wherein: The step of synthesizing the animation data of the virtual image based on the appearance resource of the virtual image, the action data and the expression data comprises: For each vertex in the skeleton skin of the avatar, determine a skin weight of each skeleton component of the avatar relative to the vertex, wherein the skin weight represents the degree to which the vertex is affected by the skeleton component; Determine the position and posture reconstruction data of each of the skeletal components based on the motion data and the expression data; Determining a vertex position of each vertex based on the pose reconstruction data and the skin weight; The animation data is synthesized based on the appearance resource and the vertex positions.

5. The method according to any one of claims 1 to 4, wherein: The method further comprises: Based on the three-dimensional model of the virtual image, deriving mesh data of the virtual image; For each of the skeletal components of the three-dimensional model, mesh data of the parts associated with the skeletal components are bound to obtain the skeletal skin of the virtual image.

6. The method according to claim 4, wherein: The step of determining the posture reconstruction data of each of the skeletal components based on the action data and the expression data comprises: Determining reconstruction key points included in each of the skeletal components of the virtual image; In a case where the reconstructed key points include skeleton key points, determining the three-dimensional pose of the reconstructed key points based on the motion data; In a case where the reconstructed key points include facial key points, determining the three-dimensional pose of the reconstructed key points based on the expression data; The three-dimensional poses of the various reconstruction key points included in the skeleton component are determined as the pose reconstruction data of the skeleton component.

7. The method according to claim 4, wherein: The step of determining the vertex position of each vertex based on the pose reconstruction data and the skin weight comprises: For each vertex of the skeleton skin, determining the associated skeleton component of the vertex from the skeleton components based on the skin weights of the skeleton components relative to the vertex; The vertex position of the vertex is determined based on the pose reconstruction data of the associated skeletal component and the skin weight.

8. The method according to any one of claims 1 to 7, wherein: The obtaining of video data of the entity object comprises: Video data of the physical object captured by an external camera; The video data is converted from a video format supported by the external camera to a preset video format, wherein the preset video format is a video format supported by the external camera. Describes the format for three-dimensional reconstruction of solid objects.

9. The method according to claim 8, wherein: The converting the video data from a video format supported by the external camera to a preset video format includes: Do at least one of the following: Starting a sub-thread for format conversion, and converting the video data from the video format supported by the external camera into the preset video format through the sub-thread; A drawing command of a graphics processor (GPU) is called, and the video data is converted from a video format supported by the external camera into the preset video format through the GPU.

10. The method according to any one of claims 1 to 9, wherein: In the case where the video data includes multiple video frames, the animation data includes multiple animation frames, wherein each animation frame is associated with a video frame, and the body movements and facial expressions of the virtual image in the animation frames match the body movements and facial expressions of the physical object in the video frames.

11. A device for generating an animation of a virtual image, the device comprising: An acquisition module configured to acquire video data of an entity object; An extraction module, configured to extract posture information of the entity object based on the video data, wherein the posture information represents a body posture and facial expression posture of the entity object presented in the video data; A reconstruction module is configured to perform three-dimensional reconstruction of the physical object based on the posture information to obtain motion data and expression data of the virtual image, wherein the motion data represents the body motion of the virtual image reconstructed based on the body posture, and the expression data represents the facial expression of the virtual image reconstructed based on the expression posture; A synthesis module is configured to synthesize animation data of the virtual image based on the appearance resources of the virtual image, the action data and the expression data, wherein the animation data represents that the virtual image wears the appearance resources, presents the facial expression and performs the body movement.

12. An electronic device comprising one or more processors and one or more memories, wherein the one or more memories store at least one computer executable instruction, and the at least one computer executable instruction is loaded and executed by the one or more processors to implement the method for generating an animation of a virtual image as described in any one of claims 1 to 10.

13. A computer-readable storage medium, wherein at least one computer-executable instruction is stored in the computer-readable storage medium, and the at least one computer-executable instruction is loaded and executed by a processor to implement the method for generating an animation of a virtual image as described in any one of claims 1 to 10.

14. A computer program product, comprising at least one computer executable instruction, wherein the at least one computer executable instruction is loaded and executed by a processor to implement the method for generating an animation of a virtual image as claimed in any one of claims 1 to 10.

Citation Information

Patent Citations

  • Live broadcast method, apparatus and system of virtual reality

    CN107277599A

  • Animation generation method, device and system and storage medium

    CN111968207A

  • Virtual character synthesis method and device, equipment and storage medium

    CN112967212A

  • Method and device for generating virtual image animation, equipment and storage medium

    CN114529639A

  • Virtual image animation generation method and device, computer equipment and storage medium

    CN116958344A

Cited By

  • UE5-based method for loading three-dimensional model during operation

    CN119417994A

  • Color dynamic mapping method and system based on unreal engine and storage medium

    CN120298510A