Three-dimensional rendering method and device, storage medium and program product

Abstract

The embodiment of the invention provides a three-dimensional rendering method and device, a storage medium and a program product. In the embodiment of the invention, a neural rendering engine is divided into a compiler and a runtime component, the compiler running on a server device compiles animation control logic and three-dimensional animation rendering logic into an animation generation model and a rendering graph respectively, and the animation generation model and the rendering graph are packaged into a resource file and issued to a terminal device. The lightweight and high-performance runtime component running on the terminal device loads and runs the resource file, generates three-dimensional animation data and executes three-dimensional rendering, so that the three-dimensional animation is displayed, the calculation amount of the terminal device is reduced, and the three-dimensional rendering efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of stereoscopic vision processing technology, and in particular to a three-dimensional rendering method, device, storage medium and program product. Background Art

[0002] In a three-dimensional (3D) rendering engine, animation generation is a crucial process. In the animation generation process, traditional 3D rendering engines usually use two methods: skeletal animation and vertex animation. Skeletal animation uses a series of bones and corresponding weights to affect the vertices of the model mesh, while vertex animation creates animation by modifying the model mesh vertex by vertex. However, these two animation technologies have high requirements on the computing resources of the terminal device. For terminal devices with limited computing resources, the rendering efficiency of the 3D rendering engine is low. Summary of the invention

[0003] Multiple aspects of the present application provide a three-dimensional rendering method, device, storage medium and program product to reduce the computational complexity of a terminal device and improve rendering efficiency.

[0004] An embodiment of the present application provides a three-dimensional rendering method, which is applicable to a runtime component in a neural rendering engine, and the runtime component runs on a terminal device. The method includes: responding to a trigger operation on a three-dimensional virtual object, obtaining a resource file of the three-dimensional virtual object, the resource file including: change control information of the three-dimensional virtual object in a time dimension, an animation generation model, and a rendering image; using the change control information as input data of the animation generation model, running the animation generation model to generate three-dimensional animation data of the three-dimensional virtual object, the three-dimensional animation data including the change state of the three-dimensional virtual object in the time dimension; performing three-dimensional rendering on the three-dimensional animation data according to the three-dimensional animation rendering logic described in the rendering image, so as to display the three-dimensional animation corresponding to the three-dimensional virtual object on a target page.

[0005] The embodiment of the present application also provides a three-dimensional data processing method, which is applicable to a compiler in a neural rendering engine, and the compiler runs on a server-side device; the method includes: in response to a display request for a three-dimensional virtual object, obtaining three-dimensional model data corresponding to the three-dimensional virtual object, the three-dimensional model data including: change control information of the three-dimensional virtual object in the time dimension, animation control logic and three-dimensional animation rendering logic; wherein the animation control logic includes: vertex data and its associated relationship, skeleton data and its associated relationship; the three-dimensional animation rendering logic includes multiple rendering nodes and a hierarchical relationship between multiple rendering nodes; the animation control logic and the three-dimensional animation rendering logic are respectively compiled into an animation generation model and a rendering image by the compiler, and the change control information is compiled into change control information in the tensor form that can be recognized by the animation generation model; a resource file is generated according to the change control information in the tensor form, the animation generation model and the rendering image, and a request end for sending a display request is sent.

[0006] An embodiment of the present application also provides an electronic device, including: a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement the steps in the three-dimensional rendering method and the three-dimensional data processing method provided in the embodiment of the present application.

[0007] An embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor implements each step of the three-dimensional rendering method and the three-dimensional data processing method provided in the embodiment of the present application.

[0008] The embodiment of the present application also provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the processor implements the steps in the three-dimensional rendering method and the three-dimensional data processing method provided in the embodiment of the present application.

[0009] In the embodiment of the present application, the neural rendering engine is divided into a compiler and a runtime component. The compiler running on the server device compiles the animation control logic and the 3D animation rendering logic into an animation generation model and a rendering image, respectively, and packages them into resource files and sends them to the terminal device. The lightweight, high-performance runtime component running on the terminal device loads and runs the resource file, generates 3D animation data and performs 3D rendering, thereby displaying 3D animation, reducing the amount of calculation of the terminal device, and improving the efficiency of 3D rendering. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0011] Figure 1 A schematic diagram of a flow chart of a three-dimensional rendering method provided for an exemplary embodiment of the present application;

[0012] Figure 2a A schematic diagram of the structure of a neural rendering engine provided for an exemplary embodiment of the present application;

[0013] Figure 2b A schematic diagram of the structure of another neural rendering engine provided for an exemplary embodiment of the present application;

[0014] Figure 3 A schematic flow chart of a three-dimensional data processing method provided for an exemplary embodiment of the present application;

[0015] Figure 4A schematic structural diagram of a three-dimensional rendering device provided for an exemplary embodiment of the present application;

[0016] Figure 5 A schematic structural diagram of an electronic device provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0018] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0019] In the embodiment of the present application, the neural rendering engine is divided into a compiler and a runtime component. The compiler running on the server device compiles the animation control logic and the 3D animation rendering logic into an animation generation model and a rendering image, respectively, and packages them into resource files and sends them to the terminal device. The lightweight, high-performance runtime component running on the terminal device loads and runs the resource file, generates 3D animation data and performs 3D rendering, thereby displaying 3D animation, reducing the amount of calculation of the terminal device, and improving the efficiency of 3D rendering.

[0020] A solution provided by an embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0021] Figure 1 A flowchart of a three-dimensional rendering method provided for an exemplary embodiment of the present application is provided. The method is applicable to a runtime component in a neural rendering engine. The runtime component runs on a terminal device, such as Figure 1 As shown, the method includes:

[0022] 101. In response to a trigger operation on a three-dimensional virtual object, a resource file of the three-dimensional virtual object is obtained, where the resource file includes: change control information of the three-dimensional virtual object in a time dimension, an animation generation model, and a rendering;

[0023] 102. Using the change control information as input data of an animation generation model, and running the animation generation model to generate three-dimensional animation data of the three-dimensional virtual object, wherein the three-dimensional animation data includes a change state of the three-dimensional virtual object in a time dimension;

[0024] 103. Perform three-dimensional rendering on the three-dimensional animation data according to the three-dimensional animation rendering logic described by the rendering image, so as to display the three-dimensional animation corresponding to the three-dimensional virtual object on the target page.

[0025] In this embodiment, a neural network rendering (Neural Network Rendering, NNR) engine is referred to as a neural rendering engine or NNR engine. The neural rendering engine divides the structure of a traditional graphics rendering engine into two parts: a compiler and a runtime component. The compiler is used to compile the animation control logic and the three-dimensional animation rendering logic into an animation generation model and a rendering image, respectively, and generates a resource file based on the animation generation model and the rendering image. The resource file can be a composite model, namely, an NNR model; the runtime component loads and runs the resource file to realize the data interaction between the animation generation model reasoning and the graphics rendering.

[0026] In this embodiment, the traditional graphics rendering engine is deployed on the terminal device, and the compiler can convert most of the codes corresponding to the traditional graphics rendering engine (codes corresponding to the animation control logic or three-dimensional animation rendering logic required for graphics rendering) into animation generation models and renderings, and convert the change control information of the three-dimensional virtual object in the time dimension into tensors that can be recognized by the animation generation model. Optionally, in the application scenarios of skeletal animation and vertex animation, the compiler can also compile skeletal data and vertex data into tensors for tensor operators. The compiler can also generate resource files that can be directly executed by the runtime component. The functions executed by the compiler do not need to be implemented on the terminal device, reducing the package size of the runtime component and achieving the purpose of lightweight. For example, the package size of the runtime component can be 2 megabytes (M). The runtime component is responsible for loading and executing the compiled resource files, managing various tasks in the rendering process, and ensuring efficient rendering performance. Through the collaborative work of the compiler and the runtime, the neural rendering engine can achieve high-quality and efficient rendering effects. Among them, the traditional graphics rendering engine does not need to be deployed entirely on the terminal device, and most of the calculation amount is realized by the compiler running on the server-side device. The runtime component has the characteristics of lightweight, fast iteration, and high performance, and can directly load and run the resource files, reducing the computing pressure of the terminal device.

[0027] Among them, the graphics rendering engine is used to convert the three-dimensional scene into a two-dimensional image. The graphics rendering engine generates images through a series of geometric transformations, lighting calculations and rasterization operations based on physical simulation and computer graphics principles. The graphics rendering engine determines the realism, smoothness and attractiveness of the two-dimensional image.

[0028] In this embodiment, a target application is running on the terminal device, and the target application can be any application that can display a three-dimensional virtual object. For example, the target application can include but is not limited to: e-commerce applications, live broadcast applications, online learning applications, social applications, etc. The runtime component can be implemented as a system function module or as an application function module of the target application, which is not limited.

[0029] In this embodiment, the three-dimensional virtual object refers to a three-dimensional geometric body created by mathematical models and algorithms in computer graphics. The three-dimensional virtual object can be a simple geometric shape, such as a cube or a sphere, or a complex model, such as a person, an animal, a building, a shoe, furniture, or a vehicle.

[0030] In this embodiment, the deformation of the virtual three-dimensional object can be controlled by skeletal animation (also called skinning animation) and vertex animation, and specifically, the deformation of the three-dimensional mesh model corresponding to the virtual three-dimensional object can be controlled. Among them, in skeletal animation, the "skeleton" in skeletal animation refers to a virtual skeleton system, which is used to control and drive the deformation of the three-dimensional mesh model. This skeleton system is composed of multiple interconnected bones, and each bone can independently perform rotation, translation and scaling operations. Among them, the connection relationship between the bones constitutes the hierarchical information between the bones. For example, if the three-dimensional virtual object is implemented as a human body, the hierarchical information of the human skeleton may include: the connection between the upper arm and the lower arm, and the connection between the head and the body. Each bone is associated with a group of vertices in the three-dimensional mesh model, and the position of the vertex is affected by the weight, thereby driving the deformation of the entire mesh and obtaining the animation data of the three-dimensional virtual object. In vertex animation, the vertices in the three-dimensional mesh model are directly changed to generate the animation data of the three-dimensional virtual object. Among them, vertex animation defines a group of vertex data for key frames, and in the process of generating animation data, each vertex data in the intermediate frame is calculated by interpolating between key frames. Vertex animation is suitable for non-bone-driven deformation effects, such as facial expressions or fluid animation.

[0031] In the field of clothing, skeletal animation can show the wearing effect of clothes and the walking action of shoes. In the field of furniture, skeletal animation can show the assembly process of furniture, the opening and closing of drawers, etc. Vertex animation can show simple mechanical movements, such as the shaking of chairs and the unfolding of tables. In the field of automobiles, skeletal animation can show the opening and closing of car doors and the rotation of wheels, etc. Vertex animation can show slight deformation of the car body, such as dents after collisions.

[0032] In this embodiment, a three-dimensional virtual object can be displayed on the target application, and the three-dimensional virtual object can be triggered, and the triggering method is not limited. For example, the user initiates a user operation on the three-dimensional virtual object to trigger the three-dimensional virtual object; for another example, the three-dimensional virtual object can be triggered by an application event, or the three-dimensional virtual object can be triggered by a system event. Among them, the user operation includes but is not limited to various interaction methods such as touch operation, gesture operation, voice operation, head movement operation, eye movement operation, etc. Touch operation includes but is not limited to: single operation, double-click operation, long press operation and sliding operation, etc. Application events may include but are not limited to: jump events or clock events from other applications to the target application, etc., and system events may include but are not limited to: power-on events, events of recovery from sleep mode, and hardware connection events, etc. Among them, the user operation can be issued through the human-machine interface provided by the terminal device, and the human-machine interface may include: command line interface (CLI), graphical user interface (GUI), text user interface (TUI), voice user interface (VUI), touch user interface (Touch UI), gesture user interface (Gestural UI), keyboard and mouse interface, natural language processing (NLP) interface, sensor and biometric interface and hardware controller. At least one of the controller.

[0033] In this embodiment, the terminal device can respond to the trigger operation for the 3D virtual object, obtain the resource file of the 3D virtual object, parse the resource file, and obtain the change control information, animation generation model and rendering graph of the 3D virtual object in the time dimension.

[0034] Among them, the change control information of the three-dimensional virtual object in the time dimension can be a series of data that changes with time, and the data expresses the translation, rotation, and scaling state of the skeleton data and vertex data in the time dimension. For example, the change control information can describe the translation, rotation, and scaling state of the skeleton or vertex of the three-dimensional virtual object in the time dimension through a curve, and the time can be continuous time or a discrete time point. The change control information can be a translation curve, a rotation curve, and a scaling curve corresponding to the skeleton A1. For example, the horizontal axis of the translation curve, the rotation curve, and the scaling curve is time, the vertical axis of the translation curve is the translation amount of the skeleton A1, the vertical axis of the rotation curve is the rotation angle of the skeleton A1, and the vertical axis of the scaling curve can be the scaling multiple of the skeleton A1. Further, the translation curve can include three types, and there are translation curves corresponding to the x-axis, y-axis, and z-axis in the three-dimensional space, respectively, which respectively represent the translation amounts in the three directions of the x-axis, y-axis, and z-axis. Further, the rotation curve can also include three types, for example, the rotation curves corresponding to the x-axis, y-axis, and z-axis respectively represent the rotation angles along the x-axis, y-axis, and z-axis respectively.

[0035] It should be noted that the change control information in the resource file is in the form of tensors that can be recognized by the animation generation model, and the change control information in the form of tensors is used as the input data of the animation generation model. Tensor is a general representation of a multidimensional array, which is used to represent and store data, such as model input data, weights, activation function output data, etc. Tensors can represent data of any dimension. For example, tensors can accommodate various data forms from single values ​​(scalars) to multidimensional data sets, such as one-dimensional arrays (vectors), two-dimensional arrays (matrices), and higher-dimensional arrays.

[0036] The reasoning process of the animation generation model reflects the data processing logic in the process of generating three-dimensional animation data. For example, the data processing logic may include a series of calculation operations that need to be performed to generate three-dimensional animation data.

[0037] For example, based on the input data, the change information of the bone B1 is determined; based on the hierarchical relationship between the bones, the change information of the bone B1 and its associated bones is calculated; based on the weight data between the bones and the vertices, the change information of the vertices corresponding to the bone B1 and its associated bones is calculated; based on the transformation information of the vertices, the three-dimensional animation data of the three-dimensional virtual object is generated. For another example, based on the input data, the vertex information of the key frames is calculated; based on the vertex information of the key frames, the vertex information of the intermediate frames between the key frames (i.e., interpolation information) is determined; based on the vertex information of the key frames and the vertex information of the intermediate frames, the three-dimensional animation data of the three-dimensional virtual object is generated.

[0038] In this embodiment, the runtime component can be used to reason about the animation generation model, that is, the change control information is used as input data of the animation generation model, and the animation generation model is run to generate three-dimensional animation data of the three-dimensional virtual object. The three-dimensional animation data includes the change state of the three-dimensional virtual object in the time dimension, such as rotation state, translation state, and scaling state.

[0039] Optionally, the animation generation model includes: an input processing sub-model, a skeleton sub-model (also called a skeleton system bonesystem), and an animation sub-model. Among them, the input processing sub-model is used to obtain the change information of each bone from the change control information; for example, the rotation, translation, and scaling state of the bone are obtained from various change curves. The skeleton sub-model maintains the hierarchical relationship between the skeleton data, the weight information between the skeleton data and the vertex data, and the interpolation weights between the vertex data. The animation sub-model is used to generate animation data based on the output data of the above two sub-models.

[0040] Among them, the animation generation model can be implemented as an NNR model, and the various sub-models included in the animation generation model can be implemented as a mobile neural network (Mobile Neural Network, MNN) model, such as a deep learning model in MNN format (for example, represented as a *.mnn file). The MNN engine is pre-deployed locally on the terminal device. The MNN engine is a lightweight, high-performance deep learning reasoning engine that can implement fast reasoning of the MNN model on the terminal device. Furthermore, in the rendering, the various sub-models of the animation generation model can be used as rendering nodes.

[0041] The rendering graph is used to organize and optimize the various rendering stages in the rendering pipeline, providing a flexible and efficient way to manage and optimize complex rendering processes. The rendering graph describes the rendering process in the form of a graph. Each rendering node in the rendering graph represents a specific rendering operation or a rendering channel. The edges in the rendering graph represent the hierarchical relationship between rendering nodes, that is, the output of the parent rendering node serves as the input of the child rendering node. The 3D animation rendering logic described by the rendering graph includes: multiple rendering nodes and the hierarchical relationship between multiple rendering nodes.

[0042] In this embodiment, the three-dimensional animation data may be three-dimensionally rendered according to the three-dimensional animation rendering logic described by the rendering image, so as to display the three-dimensional animation corresponding to the three-dimensional virtual object on the target page.

[0043] In the embodiment of the present application, the neural rendering engine is divided into a compiler and a runtime component. The compiler running on the server device compiles the animation control logic and the 3D animation rendering logic into an animation generation model and a rendering image, respectively, and packages them into resource files and sends them to the terminal device. The lightweight, high-performance runtime component running on the terminal device loads and runs the resource file, generates 3D animation data and performs 3D rendering, thereby displaying 3D animation, reducing the amount of calculation of the terminal device, and improving the efficiency of 3D rendering.

[0044] In an optional embodiment, the change control information is used as input data of the animation generation model, and the animation generation model is run to generate three-dimensional animation data of the three-dimensional virtual object. A specific implementation method is provided below. The change control information is input into the animation generation model through the scheduler corresponding to the runtime component, and the animation generation model is run; in the animation generation model, the three-dimensional animation data of the three-dimensional virtual object is generated according to the change control information in combination with the skeletal animation control logic and / or the vertex animation control logic. Among them, the skeletal animation control logic is a control logic that generates animation data based on the hierarchical information between the skeletal data and the weight information between the skeletal data and the vertex data, and the vertex animation control logic is a control logic that generates animation data based on the interpolation weights between the vertex data.

[0045] Optionally, according to the change control information, in combination with the skeletal animation control logic and / or the vertex animation control logic, the implementation method of generating the three-dimensional animation data of the three-dimensional virtual object is not limited. According to the difference of the change control information, the implementation method is also different. For example, the change control information includes: the skeletal change information of the target skeleton and / or the vertex change information of the key frame. The following is explained in detail.

[0046] Case C1, the change control information includes the bone change information of the target bone, which is applicable to bone animation. According to the bone change information of the target bone, combined with the hierarchical information between the bone data, the bone animation data of the target bone and its associated bones are generated. For example, the target bone can be an upper arm bone, and the associated bone of the upper arm bone can be a lower arm bone. According to the bone animation data of the target bone and its associated bones, combined with the weight information between the bone data and the vertex data, the vertex animation data of the corresponding vertices of the target bone and its associated bones are generated; the vertex animation data is used as the three-dimensional animation data of the three-dimensional virtual object. Among them, the weight information between the bone data and the vertex data means that the movement of the bone can affect the corresponding vertex data according to the weight. Among them, the bone data can be the position of the bone, and the vertex data can be the position of the bone.

[0047] Case C2, the change control information includes the vertex change information of the key frame, which is applicable to vertex animation. According to the vertex change information of the key frame, combined with the interpolation weights between the vertex data, the vertex data of the intermediate frame between the key frames is generated; according to the vertex data of the key frame and the vertex data of the intermediate frame, the vertex animation data is generated as the three-dimensional animation data of the three-dimensional virtual object. Among them, the higher the interpolation weight, the higher the deformation, and different facial expressions can correspond to different weights. For example, the first key frame is an image frame corresponding to the natural state of "expressionless face", and the second key frame is an image frame corresponding to the "smiling face" state. According to the interpolation weight corresponding to "smile", the intermediate frame can be inserted between the first key frame and the second key frame, and the vertex data in the intermediate frame is obtained from the vertices in the first key frame and the vertices in the second key frame based on the interpolation weight.

[0048] In case C3, the change control information includes the bone change information of the target bone and the vertex change information of the key frame. In this case, both the bone animation and the vertex animation are applicable. For example, the bone of the three-dimensional virtual object changes, and at the same time, the expression of the three-dimensional virtual object also changes. Therefore, the vertex animation data generated in case C1 can be called the first vertex animation data, and the vertex animation data generated in case C2 can be called the second vertex animation data. The first vertex animation data and the second vertex animation data are used as the three-dimensional animation data of the three-dimensional virtual object. Among them, the detailed introduction about the generation of the first vertex animation data and the second vertex animation data can be referred to the aforementioned embodiment, which will not be repeated here.

[0049] In an optional embodiment, according to the 3D animation rendering logic described by the rendering diagram, the 3D animation data is 3D rendered to display the 3D animation corresponding to the 3D virtual object on the target page. The implementation method is not limited. The following provides a specific implementation method, including:

[0050] Parse the rendering graph to obtain multiple rendering nodes and hierarchical relationships between the multiple rendering nodes;

[0051] Generate rendering pipeline description information corresponding to the three-dimensional virtual object according to the multiple rendering nodes and the hierarchical relationship between the multiple rendering nodes, the rendering pipeline description information including at least two rendering entities and their execution order, and the rendering entity including at least one component;

[0052] Converting the three-dimensional animation data into at least two entity animation data corresponding to at least two rendering entities according to the pre-maintained synchronous conversion information between the component and the tensor;

[0053] At least two entity animation data in the rendering pipeline are rendered in an execution order through a scheduler corresponding to the runtime component, so as to display the three-dimensional animation corresponding to the three-dimensional virtual object on the target page.

[0054] Each render node (Render Node) includes: a dependency relationship between it and its parent render node, a render pipeline (Render Pipeline) description, a render target (Render Target), and a render entity (Drawable). The dependency relationship between a render node and its parent render node can be understood as the output of the parent render node can be used as the input of the render node; wherein, the root render node does not have a parent render node, and the root render node does not have a dependency relationship with its parent render node. The hierarchical relationship between render nodes may include the dependency relationship between each render node and the parent render node. The render pipeline description refers to the rendering process description information of the render node, for example, the render node includes one or more render entities, and the execution order of the one or more render entities; the render target defines the output result of the render node, for example, the data format and length information of the output result. Render entities may include but are not limited to: upper arms, lower arms, lines or vertices, etc.

[0055] Among them, the neural rendering engine is described according to the traditional entity-component-system (ECS) architecture. The neural rendering engine includes entities, components and systems. Among them, an entity is a unique identifier, usually an integer ID, and an entity corresponds to a collection of components. Components are used to store data without any behavior. Components provide data representation of entities and describe certain attributes of entities. For example, position, speed, health value, rendering information, etc. Different component combinations can describe different types of entities. The system is used to process the behavior logic of entities with a specific component combination and has no related data. The system implements the behavior processing of entities by accessing and operating components. The rendering process is responsible for drawing entities with rendering components, and the system is responsible for updating the status of entities including components (such as position or speed). An entity can represent a drawable object (Drawable), that is, a rendering entity, by adding specific components. For example, an entity can add the following components to become a drawable object: 1. Transform component, including the position, rotation and scaling information of the entity. 2. Mesh Component, including the geometric data of the entity (vertices, faces, etc.). 3. Texture Component: including texture mapping information. 4. Material Component: including material properties (such as lighting parameters, color, transparency, etc.). 5. Renderable Component: marking the entity as a drawable object and providing other information required for rendering.

[0056] The implementation method of generating rendering pipeline description information corresponding to a three-dimensional virtual object according to a plurality of rendering nodes and a hierarchical relationship between the plurality of rendering nodes is not limited. For example, the rendering pipeline description information includes at least two rendering passes and their execution order, and each rendering pass includes at least one rendering entity and its execution order. The rendering pipeline description information corresponding to the three-dimensional virtual object can be generated based on the rendering pipeline description information included in each rendering node and combined with the hierarchical relationship between the plurality of rendering nodes, that is, the rendering pipeline description information of the entire rendering image dimension.

[0057] Among them, the animation generation model mainly performs model reasoning based on tensors, and in the rendering process, the entities are mainly rendered based on components. Therefore, the entire rendering process involves converting component data into tensor data, inputting the tensor data into the animation generation model to perform the reasoning process, obtaining the tensor result, and converting the tensor result into component data, and continuing the rendering process based on the converted component data. Therefore, the three-dimensional animation data can be converted into at least two entity animation data corresponding to at least two rendering entities according to the pre-maintained synchronous conversion information between the component and the tensor. Among them, the entity animation data may include at least one component animation data.

[0058] Among them, at least two entity animation data in the rendering pipeline are rendered according to the execution order in the rendering pipeline description information through the scheduler corresponding to the runtime component, so as to display the three-dimensional animation corresponding to the three-dimensional virtual object on the target page.

[0059] Optionally, the implementation method of converting the three-dimensional animation data into at least two entity animation data corresponding to at least two rendering entities according to the synchronous conversion information between the pre-maintained components and tensors is not limited. A specific implementation method is provided below. According to the synchronous conversion information between the pre-maintained components and tensors, the three-dimensional animation data is converted into multiple component animation data; according to the corresponding relationship between the components and the rendering entities, the multiple component animation data is converted into at least two entity animation data corresponding to at least two rendering entities.

[0060] In an optional embodiment, the runtime components include: a scheduler, a model engine, and a rendering pipeline. Figure 2bAs shown. Among them, the scheduler is used to schedule and control the model engine and the rendering pipeline to realize data interaction between the model engine and the rendering pipeline; the model engine is used to process the data calculation part in the animation generation model, realize the reasoning operation of the animation generation model, and generate three-dimensional animation data. The rendering pipeline is used to perform three-dimensional rendering on the three-dimensional animation data to generate three-dimensional animation. Among them, the scheduler can schedule the model engine to run the animation generation model, and the animation generation model generates three-dimensional animation data of the three-dimensional virtual object. The scheduler can schedule the rendering pipeline to perform three-dimensional rendering on the three-dimensional animation data according to the three-dimensional animation rendering logic described by the rendering diagram, so as to display the three-dimensional animation corresponding to the three-dimensional virtual object on the target page. Among them, the rendering pipeline can call the application programming interface (API) of the rendering backend, and call the underlying hardware through the API to perform three-dimensional rendering. The model engine can also call the above API to realize various computing accelerations. Among them, the application programming interface (API) can include but is not limited to: OpenGL, OpenGL ES, Metal, Vulkan and CPU, etc.

[0061] Among them, OpenGL is a cross-platform graphics API used for two-dimensional or three-dimensional graphics rendering. It supports multiple operating systems and provides a large number of functions and extensions for various graphics applications. OpenGL ES (OpenGL for Embedded Systems) is a subset of OpenGL, designed specifically for embedded systems, especially mobile terminal devices. It removes many unnecessary complexities to adapt to the resource constraints of mobile devices. Metal is a low-overhead graphics API that can interact directly with hardware to provide high-performance graphics rendering, and improves performance by reducing driver layer overhead. Vulkan is a low-overhead graphics API that provides more direct hardware control and is suitable for high-performance applications. Among them, the CPU (Central Processing Unit) is responsible for initializing the graphics API and setting up the rendering environment, including creating windows, contexts, textures, etc. The CPU can also manage the allocation and release of graphics resources, such as textures, buffers, vertex data, etc. The CPU can also send rendering commands to the GPU to control the execution order of the rendering pipeline. Among them, in Figure 2b The rendering backend APIs include OpenGL, Metal, Vulkan, and CPU as examples, but are not limited to them. Figure 2a In the figure, the runtime components include a rendering backend (Render Backend) and a model engine (such as an MNN engine) as an example for illustration, wherein the rendering backend corresponds to the rendering pipeline.

[0062] Among them, the animation generation model (for example, NNR model) contains multiple sub-models (such as MNN model), and there is also a data dependency relationship between the sub-models. The execution order of the sub-models can be determined, and the data contained in the sub-models can be distributed to each computing unit (CPU / GPU / NPU). This process is called computing scheduling. Among them, the rendering process has two independent processes: calling the model engine for reasoning and calling the rendering pipeline for rendering. These two processes have a data dependency relationship and require data interaction, which is called data synchronization.

[0063] The following is an example of the runtime component rendering process:

[0064] The runtime components include the scheduler and the render backend, where the render backend includes the model engine and the render pipeline. The scheduler is responsible for scheduling and controlling the model engine and the render pipeline, the model engine is responsible for processing the model reasoning operation of the animation generation model, and the render pipeline is responsible for 3D rendering. The render backend supports the call of the graphics API interface.

[0065] Get the resource file, parse the rendering resources, tensors, components and rendering images in the resource file. Parse the rendering targets, drawables, rendering passes and so on of different rendering nodes from the rendering image. And build the rendering pipeline description information corresponding to the rendering image and the synchronization conversion information between tensors and components. Among them, the rendering resources include: the shader code compiled and optimized by the platform, the program structure information, such as the pipeline layout, pipeline state and other structural descriptions and parameters. Among them, the shader code can be converted to the shader language of the rendering backend through the shader conversion tool, and the shader conversion tool can include but is not limited to: GLSLang, SPIRV-Cross tool. GLSLang is a tool for processing and converting shader code. SPIRV-Cross is a shader code conversion tool.

[0066] The various sub-models in the parsed animation generation model (such as the MNN model) can be calculated in an order that can be set by a channel identifier. A channel identifier is assigned to some sub-models, indicating that the sub-model is executed after the rendering of this rendering channel is completed. For sub-models that are not assigned a channel identifier, the sub-model is executed before all rendering channels are executed.

[0067] Build a runtime scheduler;

[0068] Render according to the rendering pipeline description information, where the rendering process of each frame is as follows:

[0069] According to the rendering pass construction order, the sorted rendering entities in each pass are rendered in turn. The execution content of each rendering pass includes:

[0070] The scheduler infers basic rendering operations such as MVP matrix operations, where the MVP matrix is ​​input information obtained in response to user operations, such as perspective conversion, mouse click events, etc. The MVP matrix can reflect the rotation, translation, or scaling operations performed on the three-dimensional virtual object.

[0071] Synchronize the tensors of various rendering resources (e.g., texture / cache / Uniform, etc.) or the tensors of the MNN model to the corresponding components;

[0072] Render each sorted rendering entity using the data in the above components;

[0073] Synchronize the data in each component to the corresponding tensor of the MNN model;

[0074] One or more MNN models included in the current rendering channel are inferred in sequence, and the output results are obtained into the corresponding tensor for use in the calculations involved in subsequent rendering channels (as input tensors).

[0075] Figure 3 A three-dimensional data processing method is provided for an exemplary embodiment of the present application. The method is applicable to a compiler in a neural rendering engine. The compiler runs on a server device, such as Figure 3 As shown, the method includes:

[0076] 301. In response to a display request for a 3D virtual object, obtain 3D model data corresponding to the 3D virtual object, where the 3D model data includes: change control information of the 3D virtual object in a time dimension, animation control logic, and 3D animation rendering logic; wherein the animation control logic includes: vertex data and its associated relationship, skeleton data and its associated relationship; and the 3D animation rendering logic includes multiple rendering nodes and hierarchical relationships between the multiple rendering nodes;

[0077] 302. Compile the animation control logic and the 3D animation rendering logic into an animation generation model and a rendering image respectively through a compiler, and compile the change control information into change control information in a tensor form that can be recognized by the animation generation model;

[0078] 303. Generate resource files according to the change control information in tensor form, the animation generation model and the rendering image, and send the request end of the display request.

[0079] In this embodiment, the 3D model data corresponding to the 3D virtual object includes various information for generating 3D animation of the 3D virtual object. The 3D model data can be stored and transmitted via GL Transmission Format (GLTF). Alternatively, the 3D model data can be stored and transmitted via FBX (Filmbox) format. These file formats provide a structured way to save and exchange various information of 3D models, including geometry, materials, textures, animations, etc.

[0080] Among them, GLTF is a 3D file format used to provide an efficient, compact and easy-to-parse 3D content transmission format. FBX is a 3D data exchange format that supports data transmission between multiple 3D tools and applications and is widely used in the fields of 3D animation, modeling and visual effects.

[0081] In this embodiment, the change control information in the form of tensors, the animation generation model, and the rendering can be packaged into a resource file through a serialized data format. Among them, the serialized data format refers to a format for converting data into structured data that can be stored or transmitted. FlatBuffers is an efficient serialization library, which is particularly suitable for application scenarios that require high performance and low memory overhead. By defining patterns, generating code, serializing and deserializing data, FlatBuffers provides a simple and powerful way to process structured data. The change control information, animation generation model, and rendering in the form of tensors can be converted into structured data in the FlatBuffers format, so that the runtime component does not need to parse the resource file, and can directly load and run the resource file to achieve high performance requirements.

[0082] In this embodiment, when the compiler converts the animation control logic into the animation generation model, the animation control logic is implemented by tensor operators, that is, the animation generation model can be implemented by a series of tensor operators, and the data part is implemented by tensors. The animation control logic can be implemented as a computational graph, which includes multiple computational nodes and their hierarchical relationships, wherein each computational node includes one or more computational operations.

[0083] For example, bone data and its associated relationships may include the connection relationship between bones. For example, the upper arm corresponds to a translation matrix E1. Based on the connection relationship between the upper arm and the forearm, the upper arm drives the forearm to move. The forearm corresponds to a translation matrix E2, and the forearm itself corresponds to a rotation matrix F1. Then the final motion matrix of the forearm can be the product of the translation matrix E2 and the rotation matrix F1. The connection relationship between the upper arm and the forearm can be represented by a multiplication tensor operator, and the motion matrix of the upper arm or the forearm can be represented by a tensor.

[0084] Among them, tensor operators may include but are not limited to: scalar multiplication (Scalar Multiply) operator, floor (Floor) operator, constant (const) operator, top K maximum / minimum (topK) operator, gather (gather) operator, sum (reduce sum) operator and increase dimension (unsqueeze) operator.

[0085] Among them, the scalar multiplication operator means multiplying a scalar value with each element in the tensor to adjust the size of all elements in the tensor. For example, in skeletal animation, it can be used to adjust the scaling of bones. For example, when simulating muscle contraction or expansion, the length of bones can be changed by scalar multiplication. In vertex animation, it can be used to adjust the position of vertices. For example, when simulating the effect of enlarging or shrinking an object, the coordinates of the vertex can be changed by scalar multiplication.

[0086] The floor operator represents the floor operation for each element in the tensor, that is, each element is replaced by the largest integer that is not greater than the element. It is often used in situations where floating-point numbers need to be converted to integers. For example, in skeletal animation, the rotation angle or position of the bone may need to be discretized. For example, when quantizing the angle of the bone, you can use flooring to get the nearest integer value. In vertex animation, when processing vertex positions, it is sometimes necessary to convert floating-point coordinates to integer coordinates. For example, in some rendering techniques, it may be necessary to round the vertex position down to the nearest pixel position.

[0087] The constant operator represents the creation of a tensor containing constant values. It is usually used to introduce fixed, unchanging data in the computational graph. For example, in skeletal animation, it can be used to define fixed bone parameters, such as initial position, initial rotation angle, etc. These constants can remain unchanged during the animation process as reference values. In vertex animation, it can be used to define fixed vertex attributes, such as initial position, color, etc. These constants can remain unchanged during the animation process as reference values.

[0088] The topK operator selects the top K (K is a positive integer) largest (or smallest) elements from a tensor and returns these elements and their corresponding indices. It is often used to obtain the highest ranking elements. For example, in skeletal animation, you may need to select the bones with the greatest influence. For example, when optimizing bone weights, you can select the top K bones that have the greatest influence on vertices for key adjustments. In vertex animation, you may need to select the vertices with the greatest changes. For example, when optimizing animation effects, you can select the top K vertices with the most obvious deformation for key processing.

[0089] The collection operator means collecting specific elements from a tensor according to a given index. It is often used to extract elements at specific positions from a tensor, such as obtaining corresponding values ​​according to the index returned by the TopK operation. For example, in skeletal animation, specific bone information is collected from the bone data according to the index returned by the TopK operator. For example, the rotation angles and positions of the most influential bones are collected. For another example, in vertex animation, specific vertex information is collected from the vertex data according to the index returned by the TopK operator. For example, the positions and normals of the most obviously deformed vertices are collected.

[0090] The summation operator sums the elements in the tensor along the specified dimension to calculate the sum along a certain dimension. For example, in skeletal animation, it can be used to calculate the total influence of multiple bones on a vertex. For example, when calculating the final position of a vertex, the influence of all related bones can be added up. For another example, in vertex animation, it can be used to calculate the total displacement or total deformation of a vertex. For example, when calculating the final position of a vertex, the displacement of all frames can be added up.

[0091] The add dimension operator means adding a new dimension at the specified position of the tensor, and is often used to change the shape of the tensor to meet the needs of subsequent operations. For example, in skeletal animation, in some cases, you may need to expand one-dimensional data into multi-dimensional data. For example, when processing a bone hierarchy, you may need to expand the data of a single bone into multi-dimensional data containing multiple child bones. For another example, in vertex animation, when processing vertex animation, you may need to expand one-dimensional data into multi-dimensional data. For example, when processing the color of a vertex, you may need to expand a single color value into multi-dimensional data containing multiple channels.

[0092] The compiler includes an expression model, which can compile the animation control logic (such as a computational graph) into an animation generation model through operators in the expression module. The combination of different operators reflects the bone data and its association, and the vertex data and its association. For example, the blend shape weight can be calculated through different operators to control the position change of the vertex that moves with the bone.

[0093] The expression module contains a series of expression functions that can construct all operators in MNN, and provides the type Var to describe data and expressions. Var can store intermediate calculation results and can be transferred between different expressions. Intermediate calculation results can include variables such as tensors and scalars. Through the combination of expression functions and Var variables, model reasoning, numerical calculations, and model construction of animation generation models can be realized.

[0094] In an optional embodiment, the three-dimensional model data also includes: animation control process information and rendering process information; the animation control logic and the three-dimensional animation rendering logic are compiled into an animation generation model and a rendering image respectively through a compiler, including: through a compiler, according to the animation control process information, the association relationship between vertex data and bone data is compiled into an animation generation model; according to the rendering process information, multiple rendering nodes and the hierarchical relationship between multiple rendering nodes are compiled into a rendering image.

[0095] Among them, the animation control process information and the rendering process information are related to the service logic of the target application on the terminal device, and are determined by the technical personnel or the application requirements of the target application.

[0096] The animation control process information may be description information (such as a JSON file) or may be implemented by program code. The rendering process information may be description information (such as a JSON file) or may be implemented by program code. Figure 2a In the code, the internal structure of the compiler is exemplarily shown. The compiler includes: graphics API calls, fixed parts, rendering pipeline orchestration, and artificial intelligence (AI) algorithms. Among them, the detailed introduction to the graphics API can be found in the above, which will not be repeated here. The fixed part includes but is not limited to: animation, particles, skeletons, simulation, collision, gravity and other systems. The rendering pipeline orchestration includes but is not limited to: lighting, deferred rendering shadows, ambient light, reflection and refraction, etc. AI algorithms include but are not limited to: face recognition, posture estimation and beauty, etc.

[0097] Regarding the embodiments of the present application Figure 1 and Figure 3 The detailed implementation and beneficial effects of each step in the method have been described in detail in the aforementioned embodiments and will not be elaborated here.

[0098] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 to 103 can be device A; for another example, the execution subject of steps 101 and 102 can be device A, and the execution subject of step 103 can be device B; and so on.

[0099] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0100] Figure 4 A schematic diagram of a three-dimensional rendering device provided by an exemplary embodiment of the present application is applicable to a runtime component in a neural rendering engine, and the runtime component runs on a terminal device, such as Figure 4 As shown, the device includes: an acquisition module 41, an operation module 42 and a rendering module 43;

[0101] An acquisition module, for responding to a trigger operation on a three-dimensional virtual object, and acquiring a resource file of the three-dimensional virtual object, wherein the resource file includes: change control information of the three-dimensional virtual object in a time dimension, an animation generation model, and a rendering;

[0102] An operation module is used to use the change control information as input data of the animation generation model, and to operate the animation generation model to generate three-dimensional animation data of the three-dimensional virtual object, wherein the three-dimensional animation data includes a change state of the three-dimensional virtual object in a time dimension;

[0103] The rendering module is used to perform three-dimensional rendering on the three-dimensional animation data according to the three-dimensional animation rendering logic described by the rendering image, so as to display the three-dimensional animation corresponding to the three-dimensional virtual object on the target page.

[0104] In an optional embodiment, the neural rendering engine also includes: a compiler running on the server side, the compiler being used to compile the animation control logic and the three-dimensional animation rendering logic corresponding to the three-dimensional virtual object into an animation generation model and a rendering image, respectively, and to compile the change control information into a tensor that can be recognized by the animation generation model; wherein the animation control logic includes vertex data of the three-dimensional virtual object and its associated relationships, as well as skeleton data and its associated relationships; and the three-dimensional animation rendering logic includes multiple rendering nodes and hierarchical relationships between the multiple rendering nodes.

[0105] In an optional embodiment, the running module is specifically used to: input the change control information into the animation generation model through the scheduler corresponding to the runtime component, and run the animation generation model; in the animation generation model, based on the change control information, combined with the skeletal animation control logic and / or the vertex animation control logic, generate three-dimensional animation data of the three-dimensional virtual object.

[0106] Optionally, the change control information includes: bone change information of the target bone and / or vertex change information of the key frame; the operation module is specifically used to: generate bone animation data of the target bone and its associated bones according to the bone change information of the target bone, combined with the hierarchical information between the bone data; generate vertex animation data of the vertices corresponding to the target bone and its associated bones according to the bone animation data of the target bone and its associated bones, combined with the weight information between the bone data and the vertex data; use the vertex animation data as three-dimensional animation data of a three-dimensional virtual object; and / or, generate vertex data of intermediate frames between key frames according to the vertex change information of the key frames, combined with the interpolation weights between the vertex data; generate vertex animation data as three-dimensional animation data of a three-dimensional virtual object according to the vertex data of the key frames and the vertex data of the intermediate frames.

[0107] In an optional embodiment, the rendering module is specifically used to: parse the rendering image to obtain a plurality of rendering nodes and a hierarchical relationship between the plurality of rendering nodes; generate rendering pipeline description information corresponding to the three-dimensional virtual object according to the hierarchical relationship between the plurality of rendering nodes and the plurality of rendering nodes, the rendering pipeline description information including at least two rendering entities and their execution order, the rendering entity including at least one component; convert the three-dimensional animation data into at least two entity animation data corresponding to the at least two rendering entities according to the pre-maintained synchronization conversion information between the component and the tensor; render the at least two entity animation data in the rendering pipeline according to the execution order through the scheduler corresponding to the runtime component, so as to display the three-dimensional animation corresponding to the three-dimensional virtual object on the target page.

[0108] Optionally, the running module is specifically used to: convert the three-dimensional animation data into multiple component animation data according to the pre-maintained synchronous conversion information between the components and the tensors; and convert the multiple component animation data into at least two entity animation data corresponding to at least two rendering entities according to the corresponding relationship between the components and the rendering entities.

[0109] The embodiment of the present application further provides a three-dimensional data processing device, which is applicable to a compiler in a neural rendering engine, and the compiler runs on a server device, and the device includes:

[0110] An acquisition module is used to respond to a display request for a three-dimensional virtual object and acquire three-dimensional model data corresponding to the three-dimensional virtual object, wherein the three-dimensional model data includes: change control information of the three-dimensional virtual object in a time dimension, animation control logic, and three-dimensional animation rendering logic; wherein the animation control logic includes: vertex data and its associated relationship, skeleton data and its associated relationship; and the three-dimensional animation rendering logic includes a plurality of rendering nodes and a hierarchical relationship between the plurality of rendering nodes;

[0111] A compilation module, used to compile the animation control logic and the 3D animation rendering logic into an animation generation model and a rendering image respectively through a compiler, and to compile the change control information into change control information in a tensor form that can be recognized by the animation generation model;

[0112] The generation module is used to generate resource files according to the change control information in the form of tensors, the animation generation model and the rendering image, and send the request end of the display request.

[0113] Optionally, the three-dimensional model data also includes: animation control process information and rendering process information; the compilation module is specifically used to: compile the association relationship between vertex data and bone data into an animation generation model through a compiler according to the animation control process information; compile the hierarchical relationship between multiple rendering nodes and multiple rendering nodes into a rendering graph according to the rendering process information.

[0114] Regarding the embodiments of the present application Figure 4 The detailed implementation and beneficial effects of each step in the illustrated device have been described in detail in the aforementioned embodiments and will not be elaborated on here.

[0115] Figure 5 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of the present application is applicable to a runtime component in a neural rendering engine, and the runtime component runs on a terminal device, such as Figure 5 As shown, the device includes: a memory 54 and a processor 55.

[0116] The memory 54 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method used to operate on the electronic device.

[0117] The processor 55 is coupled to the memory 54 and is used to execute the computer program in the memory 54, so as to: respond to a trigger operation on the three-dimensional virtual object, obtain a resource file of the three-dimensional virtual object, the resource file including: change control information of the three-dimensional virtual object in the time dimension, an animation generation model and a rendering; use the change control information as input data of the animation generation model, run the animation generation model to generate three-dimensional animation data of the three-dimensional virtual object, the three-dimensional animation data including the change state of the three-dimensional virtual object in the time dimension; perform three-dimensional rendering on the three-dimensional animation data according to the three-dimensional animation rendering logic described in the rendering to display the three-dimensional animation corresponding to the three-dimensional virtual object on the target page.

[0118] In an optional embodiment, the neural rendering engine further includes: a compiler running on the server, the compiler being used to compile the animation control logic and the three-dimensional animation rendering logic corresponding to the three-dimensional virtual object into an animation generation model and a rendering image, respectively, and compile the change control information into a tensor that can be recognized by the animation generation model;

[0119] Among them, the animation control logic includes vertex data of a three-dimensional virtual object and its associated relationship, as well as skeleton data and its associated relationship; the three-dimensional animation rendering logic includes multiple rendering nodes and the hierarchical relationship between the multiple rendering nodes.

[0120] In an optional embodiment, when the processor 55 uses the change control information as input data of the animation generation model and runs the animation generation model to generate three-dimensional animation data of the three-dimensional virtual object, it is specifically used to: input the change control information into the animation generation model through the scheduler corresponding to the runtime component, and run the animation generation model; in the animation generation model, generate the three-dimensional animation data of the three-dimensional virtual object according to the change control information in combination with the skeletal animation control logic and / or the vertex animation control logic.

[0121] Optionally, the change control information includes: bone change information of the target bone and / or vertex change information of the key frame; when the processor 55 generates three-dimensional animation data of the three-dimensional virtual object according to the change control information, combined with the bone animation control logic and / or the vertex animation control logic, it is specifically used to: generate bone animation data of the target bone and its associated bones according to the bone change information of the target bone, combined with the hierarchical information between the bone data; generate vertex animation data of the vertices corresponding to the target bone and its associated bones according to the bone animation data of the target bone and its associated bones, combined with the weight information between the bone data and the vertex data; use the vertex animation data as the three-dimensional animation data of the three-dimensional virtual object; and / or, generate vertex data of the intermediate frames between the key frames according to the vertex change information of the key frames, combined with the interpolation weights between the vertex data; generate vertex animation data as the three-dimensional animation data of the three-dimensional virtual object according to the vertex data of the key frames and the vertex data of the intermediate frames.

[0122] In an optional embodiment, when the processor 55 performs three-dimensional rendering on the three-dimensional animation data according to the three-dimensional animation rendering logic described in the rendering graph to display the three-dimensional animation corresponding to the three-dimensional virtual object on the target page, it is specifically used to: parse the rendering graph to obtain a plurality of rendering nodes and a hierarchical relationship between the plurality of rendering nodes; generate rendering pipeline description information corresponding to the three-dimensional virtual object according to the hierarchical relationship between the plurality of rendering nodes and the plurality of rendering nodes, the rendering pipeline description information including at least two rendering entities and their execution order, the rendering entity including at least one component; convert the three-dimensional animation data into at least two entity animation data corresponding to the at least two rendering entities according to the pre-maintained synchronization conversion information between the component and the tensor; render the at least two entity animation data in the rendering pipeline according to the execution order through the scheduler corresponding to the runtime component to display the three-dimensional animation corresponding to the three-dimensional virtual object on the target page.

[0123] Optionally, when the processor 55 converts the three-dimensional animation data into at least two entity animation data corresponding to at least two rendering entities according to the synchronous conversion information between the pre-maintained components and tensors, it is specifically used to: convert the three-dimensional animation data into multiple component animation data according to the synchronous conversion information between the pre-maintained components and tensors; convert the multiple component animation data into at least two entity animation data corresponding to at least two rendering entities according to the correspondence between the components and the rendering entities.

[0124] Regarding the embodiments of the present application Figure 5 The detailed implementation and beneficial effects of each step in the illustrated device have been described in detail in the aforementioned embodiments and will not be elaborated on here.

[0125] Further, if Figure 5 As shown, the electronic device also includes: a communication component 56, a display 57, a power component 58, an audio component 59 and other components. Figure 5 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 5 In addition, Figure 5 The components in the dashed box are optional components, not mandatory components, and the specific components depend on the product form of the electronic device. The electronic device of this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, or an IOT device, or a server device such as a conventional server, a cloud server, or a server array. If the electronic device of this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, etc., it can include Figure 5 If the electronic device of this embodiment is implemented as a server device such as a conventional server, a cloud server or a server array, it may not include Figure 5Components within the dashed box.

[0126] The embodiment of the present application also provides an electronic device, which is suitable for a compiler in a neural rendering engine, wherein the compiler runs on a server-side device, and the implementation structure of the electronic device is similar to Figure 5 The implementation structure of the electronic device shown is the same or similar, and can refer to Figure 5 The electronic device provided in this embodiment is implemented as follows: Figure 5 The difference between the electronic devices in the illustrated embodiments mainly lies in that the functions implemented by the processor executing the computer program stored in the memory are different. For the electronic device provided in this embodiment, the processor executes the computer program stored in the memory, which can be used to: in response to a display request for a three-dimensional virtual object, obtain the three-dimensional model data corresponding to the three-dimensional virtual object, the three-dimensional model data includes: change control information of the three-dimensional virtual object in the time dimension, animation control logic and three-dimensional animation rendering logic; wherein the animation control logic includes: vertex data and its associated relationship, skeleton data and its associated relationship; the three-dimensional animation rendering logic includes multiple rendering nodes and hierarchical relationships between multiple rendering nodes; the animation control logic and the three-dimensional animation rendering logic are compiled into an animation generation model and a rendering image respectively by a compiler, and the change control information is compiled into change control information in the form of a tensor that can be recognized by the animation generation model; a resource file is generated according to the change control information in the form of a tensor, the animation generation model and the rendering image, and a request end for sending a display request is generated.

[0127] In an optional embodiment, the three-dimensional model data also includes: animation control process information and rendering process information; when the processor compiles the animation control logic and the three-dimensional animation rendering logic into an animation generation model and a rendering image respectively through a compiler, it is specifically used to: compile the association relationship between vertex data and bone data into an animation generation model according to the animation control process information through the compiler; compile multiple rendering nodes and the hierarchical relationship between multiple rendering nodes into a rendering image according to the rendering process information.

[0128] The detailed implementation and beneficial effects of the electronic device provided in the embodiments of the present application have been described in detail in the aforementioned embodiments and will not be elaborated here.

[0129] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement each step that can be executed by an electronic device in the above method embodiment.

[0130] An embodiment of the present application also provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the processor is caused to implement the steps in the above method.

[0131] The above-mentioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0132] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0133] The above-mentioned display includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0134] The power supply assembly provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.

[0135] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (Microphone, MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0136] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical storage, etc.) containing computer-usable program code.

[0137] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0138] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0140] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interface, network interface and memory.

[0141] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0142] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0143] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0144] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A three-dimensional rendering method, characterized in that: Applicable to a runtime component in a neural rendering engine, the runtime component running on a terminal device, the method comprising: In response to a trigger operation on a three-dimensional virtual object, a resource file of the three-dimensional virtual object is acquired, wherein the resource file includes: change control information of the three-dimensional virtual object in a time dimension, an animation generation model, and a rendering; Using the change control information as input data of the animation generation model, and running the animation generation model to generate three-dimensional animation data of the three-dimensional virtual object, wherein the three-dimensional animation data includes a change state of the three-dimensional virtual object in a time dimension; The three-dimensional animation data is three-dimensionally rendered according to the three-dimensional animation rendering logic described by the rendering image, so as to display the three-dimensional animation corresponding to the three-dimensional virtual object on the target page.

2. The method according to claim 1, characterized in that The neural rendering engine further includes: a compiler running on the server, the compiler being used to compile the animation control logic and the three-dimensional animation rendering logic corresponding to the three-dimensional virtual object into the animation generation model and the rendering image, respectively, and compile the change control information into a tensor recognizable by the animation generation model; The animation control logic includes vertex data of the three-dimensional virtual object and its associated relationship, and skeleton data and its associated relationship; the three-dimensional animation rendering logic includes multiple rendering nodes and hierarchical relationships between the multiple rendering nodes.

3. The method according to claim 1, characterized in that The change control information is used as input data of the animation generation model, and the animation generation model is run to generate three-dimensional animation data of the three-dimensional virtual object, including: Inputting the change control information into the animation generation model through a scheduler corresponding to the runtime component, and running the animation generation model; In the animation generation model, three-dimensional animation data of the three-dimensional virtual object is generated according to the change control information in combination with skeletal animation control logic and / or vertex animation control logic.

4. The method according to claim 3, characterized in that The change control information includes: bone change information of the target bone and / or vertex change information of the key frame; generating three-dimensional animation data of the three-dimensional virtual object according to the change control information in combination with the bone animation control logic and / or the vertex animation control logic, including: Generate skeletal animation data of the target bone and its associated bones according to the skeletal change information of the target bone and in combination with the hierarchical information between the skeletal data; generate vertex animation data of the vertices corresponding to the target bone and its associated bones according to the skeletal animation data of the target bone and its associated bones and in combination with the weight information between the skeletal data and the vertex data; use the vertex animation data as the three-dimensional animation data of the three-dimensional virtual object; and / or, Based on the vertex change information of the key frames and the interpolation weights between the vertex data, the vertex data of the intermediate frames between the key frames are generated; based on the vertex data of the key frames and the vertex data of the intermediate frames, vertex animation data are generated as the three-dimensional animation data of the three-dimensional virtual object.

5. The method according to claim 1, characterized in that According to the 3D animation rendering logic described by the rendering image, the 3D animation data is 3D rendered to display the 3D animation corresponding to the 3D virtual object on the target page, including: Parsing the rendering graph to obtain a plurality of rendering nodes and hierarchical relationships between the plurality of rendering nodes; Generate rendering pipeline description information corresponding to the three-dimensional virtual object according to the multiple rendering nodes and the hierarchical relationship between the multiple rendering nodes, wherein the rendering pipeline description information includes at least two rendering entities and their execution order, and the rendering entity includes at least one component; According to the pre-maintained synchronous conversion information between the component and the tensor, the three-dimensional animation data is converted into at least two entity animation data corresponding to the at least two rendering entities; The at least two entity animation data in the rendering pipeline are rendered in the execution order through the scheduler corresponding to the runtime component, so as to display the three-dimensional animation corresponding to the three-dimensional virtual object on the target page.

6. The method according to claim 5, characterized in that According to the pre-maintained synchronous conversion information between the component and the tensor, the three-dimensional animation data is converted into at least two entity animation data corresponding to the at least two rendering entities, including: Converting the three-dimensional animation data into a plurality of component animation data according to pre-maintained synchronous conversion information between components and tensors; According to the corresponding relationship between the components and the rendering entities, the plurality of component animation data are converted into at least two entity animation data corresponding to at least two rendering entities.

7. A three-dimensional data processing method, characterized in that: A compiler applicable to a neural rendering engine, wherein the compiler runs on a server device; the method comprises: In response to a display request for a three-dimensional virtual object, obtaining three-dimensional model data corresponding to the three-dimensional virtual object, the three-dimensional model data including: change control information of the three-dimensional virtual object in a time dimension, animation control logic, and three-dimensional animation rendering logic; The animation control logic includes: vertex data and their associated relationships, skeleton data and their associated relationships; the three-dimensional animation rendering logic includes multiple rendering nodes and the hierarchical relationships between the multiple rendering nodes; Compiling the animation control logic and the three-dimensional animation rendering logic into an animation generation model and a rendering image respectively through the compiler, and compiling the change control information into change control information in a tensor form recognizable by the animation generation model; A resource file is generated according to the change control information in the tensor form, the animation generation model and the rendering image, and a request end for sending the display request is generated.

8. The method according to claim 7, characterized in that The three-dimensional model data also includes: animation control process information and rendering process information; the animation control logic and the three-dimensional animation rendering logic are compiled into an animation generation model and a rendering image respectively by the compiler, including: Through the compiler, the association relationship between the vertex data and the skeleton data is compiled into an animation generation model according to the animation control process information; and the hierarchical relationship between multiple rendering nodes and the multiple rendering nodes is compiled into a rendering graph according to the rendering process information.

9. An electronic device, characterized in that: include: Memory and processor; The memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement the steps in the method of any one of claims 1 to 6 and claims 7 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to implement the steps in the method of any one of claims 1 to 6 and claims 7 to 8.

11. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the processor is caused to implement the steps in the method of any one of claims 1-6 and claims 7-8.

Images