Unreal Engine matching-based three-dimensional virtual deformation neural network modular integration method

By integrating skeleton motion matching and lightweight neural networks in Unreal Engine, the real-time and authenticity issues of flexible body deformation in digital twin scenarios are solved, and efficient, dynamic and realistic flexible body deformation prediction is achieved, which is suitable for various digital twin environments.

CN120805664APending Publication Date: 2025-10-17NORTHWEST ELECTROMECHANICAL ENG RES INST
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510853044.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-06-24
Publication Date
2025-10-17

AI Technical Summary

Technical Problem

In existing technologies, the deformation of flexible bodies in digital twin scenarios faces the problem of difficulty in balancing real-time and authenticity. The finite element method has a large amount of calculation and is difficult to update in real time, while the rigid body binding method lacks physical laws. Existing neural network methods cannot effectively predict dynamic deformation and are difficult to integrate into engines such as Unreal Engine.

Method used

A modular integration method of 3D virtual deformation neural network based on Unreal Engine is adopted, including skeleton motion matching data extraction, flexible body deformation simulation, lightweight deformation neural network and real-time prediction module. The motion matching algorithm and lightweight neural network are combined, and dynamic deformation prediction is realized through GRU neural network and deformation residual fusion module. It is integrated into Unreal Engine using client-server communication.

Benefits of technology

It realizes efficient, dynamic and realistic prediction of flexible body deformation in Unreal Engine, reduces the cost of dataset generation, improves real-time performance and physical realism, is suitable for various digital twin environments, and has a wide range of application platforms.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120805664A_ABST
    Figure CN120805664A_ABST
Patent Text Reader

Abstract

The invention discloses a Unreal Engine matching-based three-dimensional virtual deformation neural network modular integration method, and the method comprises a skeleton action matching data extraction module which is used for randomly extracting skeleton data and role grid data of skeleton role serialization in a scene through an action matching algorithm in Unreal Engine; the flexible body deformation simulation module is used for generating flexible body dynamic deformation grid data by utilizing finite element simulation according to the role grid data; the lightweight deformation neural network module is used for training a lightweight deformation neural network according to the serialized skeleton data, role grid data and flexible body dynamic deformation grid data extracted from the Unreal Engine; and the real-time prediction module is used for predicting and rendering a dynamic deformation grid based on the lightweight deformation neural network and newly extracted skeleton data. The method can be widely applied to a high-real-time scene in the field of digital twinning, and flexible body deformation with higher reality sense can be simulated in a Unreal Engine engine in real time.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The present application relates to the field of three-dimensional virtual skeleton character flexible body deformation in digital twin software, and particularly relates to a three-dimensional virtual deformation neural network modular integration method based on Unreal Engine matching. BACKGROUND

[0002] There are a large number of three-dimensional virtual skeleton characters in the field of digital twin. Due to different body shapes and motion forms, different deformable bodies and surface-attached variable materials such as protective materials are generated, which makes the dynamic deformation of these virtual skeleton characters complex and diverse. In order to obtain a three-dimensional virtual flexible body deformation result with high physical realism, the field of computer graphics often performs triangular mesh division on the surface of the flexible body and the variable material attached thereto, and constructs a mechanical physics model, and uses the finite element method to iteratively solve the stress and spatial position of the mesh vertex. The advantage of the finite element method is that it can obtain a high-realistic mesh vertex deformation based on mechanical physics knowledge, but its disadvantage is that the calculation amount is large, and it often takes a long time to obtain the calculation result. In addition, the digital twin field has high requirements for scene instant feedback and real-time interaction, and needs to update the flexible body deformation and rendering in time. Therefore, it is particularly difficult to use the finite element method to update the spatial position coordinates of the mesh vertex in real time in the digital twin scene, and the method of rigid body binding and fixed skinning will make the deformation of the flexible body lack physical laws and have serious distortion. The digital twin scene has both real-time and realistic requirements for the deformation of the flexible body.

[0003] Combining the physical-based finite element calculation method with deep learning neural networks and using deep learning neural networks to fit the deformation process of the flexible body is a new feasible technical route to obtain a three-dimensional flexible body deformation with real-time and realism. The neural network model trained by the deformation data calculated by the finite element method has the characteristics of fast calculation speed and strong real-time compared with the mechanical physics model of the finite element method, and it can be called a lightweight deformation neural network model because it has fewer parameters. In addition, compared with the deformation calculation method of rigid body binding and fixed skinning and other non-finite element methods, the lightweight neural network model can fit the physical laws in the deformation process and express the deformation rules in the form of an implicit function, so that the deformation predicted by the neural network implicitly contains the mechanical physical laws and has a deformation prediction result with high realism similar to the finite element method.

[0004] Unreal Engine is a powerful, flexible, and highly customizable game engine developed by Epic Games, widely used in electronic game development, virtual reality, and augmented reality application development. Unreal Engine has a wide range of applications in digital twins and digital games. In the field of digital twins, Unreal Engine can be used to create highly realistic virtual buildings and city environments for architectural design, planning, and visualization, and can also be used to simulate factories, equipment, and processes. In the field of digital games, Unreal Engine is known for its excellent graphics rendering and animation system, allowing game developers to create more realistic visual effects.

[0005] Action matching search algorithm is an animation algorithm for adding animation data to virtual characters, and unlike traditional animation systems, it does not require manual setting of complex state machines, but uses pose sequences in the animation database to dynamically match the current character pose and expected trajectory to drive the animation of virtual characters, achieving smooth transition and seamless integration.

[0006] The method of combining finite element method and deep learning neural network to obtain real-time and real flexible body deformation still has many limitations: (1) The existing method of using neural network to fit and predict flexible body deformation can only predict the static and quasi-static deformation of flexible body, without considering the influence of dynamic factors, resulting in the lack of dynamic effect of the final predicted deformation.

[0007] (2) The existing neural network method mostly uses motion capture data as the skeleton pose dataset and makes corresponding flexible body deformation data based on it. The number of motion capture sequences in this method is limited, and the production cost and process are extremely cumbersome, making it difficult to generate a large dataset in batches.

[0008] (3) The current flexible body deformation prediction neural network method still has a certain distance from direct application to the field of digital twins. The trained lightweight deformation neural network is mostly implemented using open source neural network frameworks, which is difficult to directly integrate into engines such as Unreal Engine. SUMMARY

[0009] The purpose of the present application is to provide a three-dimensional virtual deformation neural network modular integration method based on Unreal Engine matching, to solve the problems in the prior art.

[0010] In order to achieve the above tasks, the present application adopts the following technical solutions: The three-dimensional virtual deformation neural network modular integration method based on Unreal Engine matching comprises: The skeleton action matching data extraction module is configured to extract skeleton data and character mesh data of a serialized skeleton character in a scene in Unreal Engine by using an action matching algorithm. The flexible body deformation simulation module is configured to generate flexible body dynamic deformation mesh data by using finite element simulation according to the character mesh data. The lightweight deformation neural network module is configured to train a lightweight deformation neural network according to the extracted serialized skeleton data, character mesh data and flexible body dynamic deformation mesh data in Unreal Engine. The real-time prediction module is configured to, after extracting skeleton data and character mesh data of a serialized skeleton character to be predicted by using the skeleton action matching data extraction module during actual operation prediction, perform prediction of a dynamic deformation mesh of the skeleton character by using the trained lightweight deformation neural network, and perform dynamic deformation rendering in an Unreal Engine scene.

[0011] Further, the skeleton character is controlled to randomly roam in a scene built by Unreal Engine by random input through an external keyboard or controller, and skeleton data and character mesh data of the skeleton character are extracted from the scene at a preset frequency in the form of an interface; the skeleton data includes transformation matrices of each skeleton joint at a current time frame; and the character mesh data includes spatial position coordinates of mesh nodes.

[0012] Further, an action database in Unreal Engine includes action sequences corresponding to a plurality of action key frames of a skeleton character, each action key frame including spatial position and rotation information of a skeleton joint, and the information is stored in the form of a vector; in the action matching algorithm, smooth speed and smooth orientation are combined to search for a matching action sequence that matches a current external input by using a query feature vector, so as to extract skeleton data and character mesh data.

[0013] Further, the lightweight deformation neural network is composed of a dynamic skeleton encoder module and a deformation residual fusion module; after the serialized skeleton data is input into the dynamic skeleton encoder module to obtain serialized skeleton dynamic information, the serialized skeleton dynamic information is input into the deformation residual fusion module to obtain a dynamic deformation residual component. The extracted character mesh data is added to the dynamic deformation residual component, and a dynamic deformation mesh of the skeleton character is obtained by LBS skinning operation.

[0014] Furthermore, the dynamic skeleton encoder module is composed of a GRU neural network module, a regularization layer, and an activation function connected in series and repeated four times, and then connected to a GRU neural network module; the GRU neural network module is used to extract features from the input serialized skeleton data to obtain the hidden information at the current moment; the regularization layer is used to regularize the extracted hidden information; the activation function layer is used to perform nonlinear transformation on the regularized hidden information.

[0015] Furthermore, the skeleton data serialized at the current time t includes the transformation matrix of the skeleton joints; when it is input to the dynamic skeleton encoder module of the lightweight deformation neural network, it is flattened into a one-dimensional transformation vector ; Hidden information at time t-1 It is the hidden information output by the GRU neural network module in the dynamic skeleton encoder module at the previous moment; At time t, the transformation vector and hidden information Perform fusion and obtain the GRU neural network module update gate based on the weight matrix of the update gate , based on the weight matrix of the reset gate, the reset gate is obtained ; The current new memory content of the GRU neural network module Calculated using a weight matrix based on memory content and a reset gate; The GRU neural network module finally memorizes the content, that is, the hidden information Will use the update gate To decide, expressed as ;in Represents an element-wise product operation.

[0016] Furthermore, the deformation residual fusion module includes: The latent vector expression unit is used to convert the time-related dynamic features output by the dynamic skeleton encoder module into latent vectors of feature expression in the latent space; the matrix fusion unit uses a trainable grid matrix group as the fusion matrix and fuses it with the extracted latent vector to obtain the dynamic deformation residual component.

[0017] Furthermore, the trainable grid matrix group is expressed as ,in Indicates the k A grid matrix, k Represents the dimension of the latent vector; for the matrix , which is expressed as follows:

[0018] in, represents the trainable parameters of the neural network, Indicates the number of vertices of the character mesh; ; The fusion operation in the matrix fusion unit is expressed as follows:

[0019] in, Represents the dynamic deformation residual component obtained after fusion, represents the first j feature vectors.

[0020] Furthermore, the extracted character mesh data is added with the dynamic deformation residual component, and the dynamic deformation mesh of the skeleton character is obtained through the LBS skinning operation, which is expressed as: in, Represents the character mesh in its initial state, Represents the extracted character mesh data after corrections have been made to the serialized skeleton data. Represents the dynamic deformation residual component obtained by the deformation residual fusion module; Indicates the current t The transformation vector of the transformation matrix of the skeleton joint contained in the skeleton data serialized at the moment frame, Dynamic Skeleton Encoder Module Representing Lightweight Morphablet Neural Networks t -1 moment of hidden information, A dynamically deforming mesh representing the skeletal character at the target pose, Represents the skeleton structure of the skeleton character, represents the trainable skin weight matrix, Represents a linear skinning function.

[0021] Furthermore, the training of lightweight deformable neural network adopts the following loss function:

[0022] in, Represents a mesh node in a dynamically deforming mesh predicted by a lightweight deformable neural network The spatial position coordinates of Represents mesh nodes in the dynamic deformation mesh data of a flexible body The spatial position coordinates of Indicates the number of grid nodes, express Distance calculation method.

[0023] Further, the interaction between Unreal Engine and the lightweight morphable neural network is carried out in the manner of client and server communication; wherein the lightweight morphable neural network runs on the server in a listening manner, and the Unreal Engine sends skeleton data and mesh data to the lightweight morphable neural network on the server side as a client and receives predicted dynamic deformation mesh data.

[0024] A terminal device comprising a processor, a memory, and a computer program stored in the memory; when the processor executes the computer program, the three-dimensional virtual morphable neural network modular integration method based on Unreal Engine matching is realized.

[0025] A computer readable storage medium, the medium stores a computer program; when the computer program is executed by a processor, the three-dimensional virtual morphable neural network modular integration method based on Unreal Engine matching is realized.

[0026] Compared with the prior art, the present application has the following technical features: 1. Compared with the existing finite element method and deep learning neural network combined method, the present application can utilize the time sequence related skeleton dynamic information to make the prediction result of the lightweight morphable neural network have obvious dynamic characteristics; the lightweight morphable neural network in the present application utilizes a dynamic skeleton encoder module and a deformation residual fusion module, which can make the model parameter amount very small.

[0027] 2. Compared with the existing flexible body deformation method in the Unreal Engine engine, the predicted flexible body dynamic deformation result in the present application can simultaneously maintain high physical realism and good real-time performance; the present application combines the action matching algorithm and the lightweight neural network, which reduces the cost of generating a large amount of action sequence data set, and can conveniently realize actual application integration; the present application utilizes the Unreal Engine engine and the lightweight neural network, proposes four modular structures, and connects these modules through the training process and the runtime prediction process, realizes the separation of neural network training and loading running, and the modular structure makes the application range of the present application wider. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 It is a general modular integration flowchart of the present application.

[0029] Figure 2 It is a framework schematic diagram of the training process of the three-dimensional virtual morphable neural network modular integration method based on Unreal Engine action matching according to the embodiment of the present application.

[0030] Figure 3A schematic diagram of a framework for runtime prediction process of a three-dimensional virtual morphing neural network modular integration method based on Unreal Engine motion matching according to an embodiment of the present application.

[0031] Figure 4 A schematic diagram of a data interaction mode between a lightweight morphing neural network based on serialized skeleton motion matching and an Unreal Engine scene according to an embodiment of the present application. DETAILED DESCRIPTION

[0032] The three-dimensional virtual morphing neural network modular integration method based on Unreal Engine matching can combine the motion matching algorithm in Unreal Engine and the lightweight morphing neural network, and is conveniently applied to existing engines. The method is formulated as follows:

[0033] wherein, represents the motion matching algorithm in Unreal Engine, represents the current time frame, t is an algorithm parameter, represents the character mesh in the initial state, represents the skin weight matrix, represents the lightweight morphing neural network integrated in Unreal Engine, represents the loaded dynamic morphing mesh in Unreal Engine.

[0034] Referring to the accompanying drawings,The three-dimensional virtual morphing neural network modular integration method based on Unreal Engine matching provided by the present application comprises: Figure 1 1. A skeleton motion matching data extraction module for extracting skeleton data and character mesh data of a skeleton character serialized in a scene in Unreal Engine using a motion matching algorithm.

[0035] Specifically, the skeleton character randomly roams in a scene built by Unreal Engine using a motion matching algorithm, and extracts skeleton data and character mesh data of the skeleton character serialized in the scene in the form of an interface at a frequency of 25 FPS. The skeleton data includes transformation matrices of each skeleton joint at the current time frame, and the character mesh data includes spatial position coordinates of mesh nodes.

[0036] 1.1 Motion database in Unreal Engine.

[0037] ​The motion database in Unreal Engine contains a plurality of motion sequences corresponding to a plurality of motion key frames of a skeletal character, each motion key frame including spatial position and rotation information of a skeletal joint, which are stored in the form of vectors, and a motion matching algorithm in Unreal Engine constructs a feature vector for each pose in the motion key frame and is expressed by the following formula:

[0038] wherein, and denote the feature vector (spatial position) at the current pose, and denote the trajectory feature vector (rotation information) of the skeletal character; specifically, denote the velocity vector at the skeletal foot and root joint, denote the position coordinates of the skeletal foot joint, and denote the two-dimensional position coordinates and the two-dimensional orientation vector of the skeletal character at 0.33 seconds, 0.66 seconds and 1 second in the future, respectively.

[0039] 1.2 Motion matching algorithm.

[0040] In the motion matching algorithm, the smooth velocity and the smooth orientation are used to search for a suitable motion sequence, and the smooth velocity is expressed by the following formula:

[0041] wherein, denotes the velocity at the current time, denotes the velocity at the next time, denotes the time step, denotes a response factor for adjusting the speed from to , denotes the user input velocity of the external keyboard or controller hardware.

[0042] The smooth orientation is expressed by the following formula:

[0043] wherein, denotes the orientation vector at the current time, denotes the orientation vector at the next time, denotes a spherical linear interpolation function, denotes the user input orientation vector of the external keyboard or controller hardware.

[0044] In the action matching algorithm, the query feature vector is used to search for the most matching action sequence in the action database; before each search begins, the action matching algorithm first makes the query feature vector consistent with the feature vector of the current time frame, and then updates the query feature vector using the following formula:

[0045] wherein, represents the target position of the skeleton character, which is calculated by projecting the center position of the head of the character to the ground, represents the orientation vector at the n+20 time point (i.e., 0.33 seconds in the future), represents the vector in the query feature vector corresponding to , and and both contain the position and orientation information 0.33 seconds, 0.66 seconds and 1 second in the future, which are predicted.

[0046] Therefore, the action sequence suitable for the current external input can be searched in the action database according to the query feature vector. A large number of character action sequences can be conveniently obtained by randomly inputting through an external keyboard or controller to control the skeleton character to roam in the Unreal Engine scene; the skeleton action matching data extraction module extracts the serialized skeleton data and character mesh data in real time at a frequency of 25 FPS using the action matching algorithm.

[0047] It should be noted that the skeleton action matching data extraction module only stores the spatial position and rotation information of the skeleton joints during actual operation, and the character mesh data is generated by calling the local spatial position and rotation information after the extraction process is completed. The purpose of this operation is to prevent the saving of large character mesh data from affecting the extraction frequency.

[0048] 2. A flexible body deformation simulation module for generating flexible body dynamic deformation mesh data using finite element simulation based on the character mesh data; this module can be specifically implemented using Houdini design simulation software.

[0049] In order to overcome the problems of high production difficulty and complex process in obtaining the action capture method for training data set of neural network, in this scheme, the flexible body deformation simulation module is combined with the skeleton action matching data extraction module to randomly generate a large number of skeleton character motion sequences based on Unreal Engine, and an API interface for extracting skeleton data and character mesh data is constructed to generate a large amount of skeleton data and character mesh data for subsequent training of lightweight deformation neural network.

[0050] 3. A lightweight morphing neural network module for training a lightweight morphing neural network according to serialized skeleton data and flexible body dynamic morphing mesh data extracted in Unreal Engine.

[0051] In order to overcome the problem that the existing morphing neural network can only predict static and quasi-static deformation, the present application provides a lightweight morphing neural network, which aims to use serialized skeleton data to change the deformation of the flexible body from static and quasi-static to dynamic; the serialized skeleton data is affected by time frames, including information of the last time frame and data of the current time frame, which can capture variable data changing over time, so as to add the variable to the neural network, and change the static and quasi-static deformation prediction lightweight neural network to a dynamic deformation prediction lightweight neural network. The lightweight morphing neural network module proposed in the present application adds time sequence variable factors affecting dynamic deformation to the skeleton data, which can get rid of the dependence of the dynamic deformation prediction lightweight neural network on the engine and various integrated environments, and can be conveniently integrated into various digital twin field engines in a modular form.

[0052] 3.1 Lightweight morphing neural network.

[0053] The lightweight morphing neural network is composed of a dynamic skeleton encoder module and a deformation residual fusion module; after the serialized skeleton data is input into the dynamic skeleton encoder module to obtain serialized skeleton dynamic information, the serialized skeleton dynamic information is input into the deformation residual fusion module to obtain a dynamic deformation residual component; The extracted character mesh data is added to the dynamic deformation residual component, and a dynamic deformation mesh of the skeleton character is obtained through LBS skinning operation.

[0054] 3.1.1 Dynamic skeleton encoder module.

[0055] The dynamic skeleton encoder module is composed of a GRU neural network module, a regularization (LayerNorm) layer, an activation function (PReLU) in series and repeated four times, and then connected with a GRU neural network module, wherein: The GRU neural network module is used for feature extraction from the input serialized skeleton data to capture time sequence related dynamic features and obtain hidden information at the current time; The regularization (LayerNorm) layer is used for regularization operation on the extracted hidden information, which can reduce the learning difficulty of the neural network and accelerate the convergence effect; The activation function (PReLU) layer is used for nonlinear transformation of the regularized hidden information to obtain serialized skeleton dynamic information; the activation function layer aims to improve the fitting ability of the neural network and add more nonlinearity to the dynamic morphing lightweight neural network.

[0056] (1) The skeleton data of the current time t includes the transformation matrix of the skeleton joint; when input into the dynamic skeleton encoder module of the lightweight deformation neural network, it is flattened into a one-dimensional transformation vector , which is expressed as follows:

[0057] wherein represents the vector corresponding to the nth transformation matrix, and in the embodiment .

[0058] Hidden information at time t-1 is the hidden information output by the GRU neural network module in the dynamic skeleton encoder module at the previous time, and the initial value is a zero vector.

[0059] (2) In order to extract features from the serialized skeleton data, the application adopts a GRU neural network module to process the serialized skeleton data; specifically, taking the first GRU neural network module as an example: taking the transformation vector at the current time and the hidden information at the previous time as inputs, the time sequence related skeleton information can be extracted.

[0060] At time t, the transformation vector and the hidden information are fused to obtain the state of the update gate of the GRU neural network module, and the calculation formula is as follows:

[0061] wherein represents an activation function, is the weight matrix of the update gate.

[0062] The reset gate of the GRU neural network module is calculated as follows:

[0063] wherein is the weight matrix of the reset gate.

[0064] The new memory content of the GRU neural network module at the current time will store the past related information using the reset gate, and the calculation method is as follows:

[0065] wherein represents the weight matrix related to the current memory content of the GRU neural network module, which is obtained by training; represents the hyperbolic tangent activation function, represents an element-wise multiplication operation.

[0066] (3) The final memory content of the GRU neural network module, i.e., the hidden information The update gate will be used to determine the current memory content and the hidden information of the previous time which needs to be collected, and the calculation formula is as follows:

[0067] Thus, the current time hidden information of the output of the GRU neural network module can be obtained .

[0068] The current time hidden information of the output of the GRU neural network module will be input into the regularization (LayerNorm) layer, the activation function (PReLU) layer; and then output to the next GRU neural network module, the regularization (LayerNorm) layer, the activation function (PReLU) layer, until the last GRU neural network module outputs the captured time-related dynamic features.

[0069] 3.1.2 Deformation residual fusion module.

[0070] The structure of the deformation residual fusion module includes: a hidden vector expression unit, configured to convert the time-related dynamic features output by the dynamic skeleton encoder module into a hidden vector of feature expression in a hidden space ; wherein, represents the i-th feature vector, j , represents the dimension of the time-related dynamic features. k

[0071] a matrix fusion unit, which adopts a set of trainable grid matrices as a fusion matrix, and fuses the extracted hidden vector to obtain a dynamic deformation residual component.

[0072] Specifically, the set of trainable grid matrices is represented as , wherein represents the k-th grid matrix, and k represents the dimension of the hidden vector ; for the matrix among them, the formula is as follows:

[0073] wherein, represents a neural network trainable parameter, represents the number of vertices of the role grid. ​

[0074] The fusion operation in the matrix fusion unit is expressed by the following formula:

[0075] wherein, represents the dynamic deformation residual component obtained after fusion.

[0076] 3.2 LBS skinning operation.

[0077] The LBS skinning operation is a time sequence related dynamic skinning model calculation, and the specific formula is as follows:

[0078]

[0079] wherein, represents the role mesh in the initial state, represents the role mesh data corrected under the influence of the sequenced skeleton data (i.e. the extracted role mesh data), represents the dynamic deformation residual component obtained by the deformation residual fusion module; represents the current t transform vector of the transform matrix of the skeleton joint contained in the frame sequenced skeleton data at the time t, represents the dynamic skeleton encoder module of the lightweight deformation neural network t hidden information at time t-1, represents the dynamic deformation mesh of the skeleton role under the target pose, represents the skeleton structure of the skeleton role, represents the trainable skinning weight matrix, represents the linear skinning function.

[0080] In order to better capture the dynamic characteristics of the flexible body deformation, the present application uses a set of trainable skinning weight residual matrix groups to generate weight increments, and m represents the number of skinning weight residual matrices; for , , the following formula can be used:

[0081] wherein, represents the neural network parameter, is the number of skeleton joints, represents the number of vertices of the role mesh.

[0082] The skinning weight increment is expressed by the following formula:

[0083] The final skin weight matrix is calculated as follows:

[0084] wherein, The skin weight matrix of the nearest point searched from the character mesh through the KD tree structure is taken as the initial weight.

[0085] 3.3 Training of the lightweight morphing neural network.

[0086] Figure 1 The solid arrow part in the figure indicates the training process of the lightweight morphing neural network in Unreal Engine, and the dashed arrow part indicates the runtime prediction process of the lightweight morphing neural network in Unreal Engine.

[0087] In the training process, the skeleton action matching data extraction module in Unreal Engine outputs two parts of serialized skeleton data and character mesh data, wherein the serialized skeleton data is input into the lightweight morphing neural network module, and the character mesh data is input into the flexible body deformation simulation module; the flexible body deformation simulation module uses Houdini designed simulation software as a tool to obtain dynamic deformation; the lightweight morphing neural network module uses the serialized skeleton data, the character mesh data and the flexible body dynamic deformation mesh data obtained by simulation as training data to train the lightweight morphing neural network.

[0088] The training of the lightweight morphing neural network adopts the following loss function:

[0089] wherein, represents the spatial position coordinates of the grid node in the dynamic deformation mesh predicted by the lightweight morphing neural network, represents the spatial position coordinates of the grid node in the flexible body dynamic deformation mesh data obtained by simulation of the flexible body deformation simulation module, represents the number of grid nodes, represents the distance calculation method.

[0090] In order to overcome the problem that the current flexible body deformation prediction neural network method is difficult to apply to the field of digital twinning, the training process and the prediction process of the lightweight morphing neural network are independently carried out, and they are integrated into the Unreal Engine to realize the integrated application of the prediction neural network.

[0091] 4. Real-time prediction module, for loading the trained lightweight morphing neural network in Unreal Engine to predict the dynamic morphing mesh of the skeletal character after the skeletal data and character mesh data of the skeletal character sequence to be predicted are extracted by the skeletal motion matching data extraction module during actual operation prediction, and performing dynamic morphing rendering in the Unreal Engine scene.

[0092] Wherein, the interaction between Unreal Engine and lightweight morphing neural network is carried out in the way of client and server communication; wherein the lightweight morphing neural network runs on the server in the way of listening, and Unreal Engine sends the skeletal data and mesh data to the lightweight morphing neural network on the server side as a client and receives the predicted dynamic morphing mesh data.

[0093] The modular integration method proposed in the application can be seamlessly integrated into Unreal Engine engine, and can be independently run, can be linked with other functional modules of Unreal Engine engine, and the modular integration method is not dependent on the scene and can be applied to various digital twin environments developed by Unreal Engine engine.

[0094] Embodiment: In the embodiment of the application, Houdini design simulation software is used to obtain the training data required for training the lightweight morphing neural network; specifically, the character mesh data extracted by the skeletal motion matching data extraction module in Unreal Engine is stored in the form of FBX file as the action sequence generated by random roaming. Load the FBX action sequence file in Houdini, and design the flexible body mesh in the initial posture state; then set the action play frequency of FBX to 25FPS in Houdini, and start the simulation of the flexible body dynamic morphing result on the action sequence; after the simulation is completed, store the deformed sequence in the FBX file.

[0095] Then extract the flexible body deformation mesh from the deformed sequence FBX file frame by frame; in the training process, the skeletal data, character mesh data and corresponding flexible body dynamic morphing mesh data of the sequence frame are combined to form a set of training data, and the truncated backpropagation through time algorithm (TBPTT) is used to make the network converge; the number of forward time frames calculated by the truncated backpropagation through time algorithm used in the application is 100.

[0096] The lightweight morphing neural network training process adopts Adam optimizer to minimize the loss function, and the learning rate in the training process is 1E-3; the lightweight morphing neural network training needs to spend more than twenty hours.

[0097] The running time prediction process of the embodiment is as shown in the figure Figure 3 As shown in the figure, the external keyboard or controller inputs the target direction, the skeleton data at time t-1 is searched in the action database in Unreal Engine according to the action matching algorithm to obtain the skeleton data and the character mesh data at time t, and then input into the lightweight morphing neural network to predict the dynamic deformation residual component in the posture in real time; the character mesh in the standard posture is added with the dynamic deformation residual component, and the skinning operation is performed to obtain the final deformed flexible body dynamic deformation mesh; after the skeleton character mesh is redirected in Unreal Engine, the deformed flexible body mesh is rendered in real time in the Unreal Engine scene.

[0098] As shown in the figure Figure 4 In the present application, the lightweight morphing neural network loaded in Unreal Engine is completed by the communication between the server and the client. This method can avoid the model conversion operation responsible for the training and deployment of the lightweight neural network, so as to avoid the network structure mismatch and precision loss problems caused by model conversion. Specifically, the lightweight morphing neural network runs on the server side and is in a listening state, and the three-dimensional scene in Unreal Engine is used as a client to send and receive data. When starting to run, the Unreal Engine scene sends the serialized skeleton data to the server side, the lightweight morphing neural network on the server side listens to the sent data, performs prediction and returns the predicted data to the client Unreal Engine scene, and the Unreal Engine scene calculates the new deformation after receiving the returned data and performs real-time rendering in the scene.

[0099] In the present embodiment, the communication between the Unreal Engine scene client and the lightweight morphing neural network server adopts the Socket socket mode, and the data transmission adopts the HTTP protocol.

[0100] The method of integrating the lightweight morphing neural network in the Unreal Engine scene proposed by the present application can be switched according to the hardware state; specifically, the lightweight morphing neural network is preferentially run on the GPU hardware, and if the runtime does not have GPU hardware, the network can be run on the CPU hardware. The present application tests the running time of the integration method on the GPU hardware platform of NVIDIA GeForce RTX3090 and the CPU hardware platform of Intel Core i7-7700, and the running time is shown in Table 1.

[0101] Table 1 Running time

[0102] The parameter size of the lightweight deformation neural network model proposed in the application is shown in Table 2.

[0103] Table 2 Network model parameters

[0104] It can be seen through tests that the three-dimensional virtual deformation neural network modular integration method based on Unreal Engine action matching has the following advantages: Compared with fixed skinning or animation libraries in Unreal Engine, the method of the application can provide results with stronger realism and more diversified and dynamic deformation, while maintaining good real-time performance, and is a feasible method for simultaneously obtaining the realism and real-time performance of flexible body deformation results.

[0105] Compared with other methods, the application can run on a GPU or a CPU according to a hardware state, and has a wider application platform.

[0106] The application uses serialized information, and the predicted deformation results have more dynamic effects and are more reliable.

[0107] The above examples are only used to illustrate the technical solutions of the application, and not to limit them; although the application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can still be modified, or some technical features can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the application, and should be included in the protection scope of the application.

Claims

1. A modular integration method for 3D virtual morphing neural networks based on Unreal Engine matching, characterized by: include: The skeleton action matching data extraction module is used to randomly extract the skeleton data and character mesh data of the skeleton character in the scene using the action matching algorithm in Unreal Engine; Flexible body deformation simulation module, used to generate flexible body dynamic deformation mesh data using finite element simulation based on character mesh data; A lightweight morphable neural network module, which is used to train a lightweight morphable neural network based on serialized skeleton data, character mesh data, and flexible body dynamic deformation mesh data extracted from Unreal Engine; The real-time prediction module is used to extract the serialized skeleton data and character mesh data of the skeleton character to be predicted using the skeleton action matching data extraction module during the actual prediction operation, and then use the trained lightweight deformation neural network to predict the dynamic deformation mesh of the skeleton character and perform dynamic deformation rendering in the Unreal Engine scene.

2. The modular integration method of 3D virtual morphing neural network based on Unreal Engine matching according to claim 1, characterized in that: The skeleton character is controlled by random input from an external keyboard or controller, and randomly roams in the scene built by Unreal Engine using an action matching algorithm. The skeleton character's serialized skeleton data and character mesh data are extracted from the scene at a preset frequency through an interface. The skeleton data includes the transformation matrix of each skeleton joint in the current frame; the character mesh data includes the spatial position coordinates of the mesh nodes.

3. The modular integration method of 3D virtual morphing neural network based on Unreal Engine matching according to claim 1, characterized in that: The action database in Unreal Engine contains action sequences corresponding to multiple action keyframes of the skeleton character. Each action keyframe includes the spatial position and rotation information of the skeleton joints, which are stored in the form of vectors. The action matching algorithm uses smooth speed and smooth orientation combined with the query feature vector to search for an action sequence that matches the current external input, thereby extracting skeleton data and character mesh data.

4. The modular integration method of 3D virtual morphing neural network based on Unreal Engine matching according to claim 1, characterized in that: The lightweight deformable neural network consists of a dynamic skeleton encoder module and a deformation residual fusion module. The serialized skeleton data is input into the dynamic skeleton encoder module to obtain the serialized skeleton dynamic information, which is then input into the deformation residual fusion module to obtain the dynamic deformation residual component. The extracted character mesh data is added with the dynamic deformation residual component, and the dynamic deformation mesh of the skeleton character is obtained through LBS skinning operation.

5. The modular integration method of 3D virtual morphing neural network based on Unreal Engine matching according to claim 1, characterized in that: The dynamic skeleton encoder module consists of a GRU neural network module, a regularization layer, and an activation function connected in series four times, followed by another GRU neural network module. The GRU neural network module is used to extract features from the input serialized skeleton data to obtain the hidden information at the current moment. The regularization layer is used to regularize the extracted hidden information. The activation function layer is used to perform nonlinear transformation on the regularized hidden information.

6. The modular integration method of 3D virtual morphing neural network based on Unreal Engine matching according to claim 1, characterized in that: The serialized skeleton data at the current time t includes the transformation matrix of the skeleton joints; when it is input to the dynamic skeleton encoder module of the lightweight deformation neural network, it is flattened into a one-dimensional transformation vector ; Hidden information at time t-1 It is the hidden information output by the GRU neural network module in the dynamic skeleton encoder module at the previous moment; At time t, the transformation vector and hidden information Perform fusion and obtain the GRU neural network module update gate based on the weight matrix of the update gate , based on the weight matrix of the reset gate, the reset gate is obtained ; The current new memory content of the GRU neural network module Calculated using a weight matrix based on memory content and a reset gate; The GRU neural network module finally memorizes the content, that is, the hidden information Will use the update gate To decide, expressed as ;in Represents an element-wise product operation.

7. The modular integration method of 3D virtual morphing neural network based on Unreal Engine matching according to claim 1, characterized in that: The deformation residual fusion module includes: The latent vector expression unit is used to convert the time-related dynamic features output by the dynamic skeleton encoder module into latent vectors of feature expression in the latent space; the matrix fusion unit uses a trainable grid matrix group as the fusion matrix and fuses it with the extracted latent vector to obtain the dynamic deformation residual component. Furthermore, the trainable grid matrix group is expressed as ,in Indicates the k A grid matrix, k Represents the dimension of the latent vector; for the matrix , which is expressed as follows: in, represents the trainable parameters of the neural network, Indicates the number of vertices of the character mesh; ; The fusion operation in the matrix fusion unit is expressed as follows: in, Represents the dynamic deformation residual component obtained after fusion, represents the first j feature vectors.

8. The modular integration method of 3D virtual morphing neural network based on Unreal Engine matching according to claim 1, characterized in that: The extracted character mesh data is added with the dynamic deformation residual component, and the dynamic deformation mesh of the skeleton character is obtained through the LBS skinning operation, which is expressed as: in, Represents the character mesh in its initial state, Represents the extracted character mesh data after corrections have been made to the serialized skeleton data. Represents the dynamic deformation residual component obtained by the deformation residual fusion module; Indicates the current t The transformation vector of the transformation matrix of the skeleton joint contained in the skeleton data serialized at the moment frame, Dynamic Skeleton Encoder Module Representing Lightweight Morphablet Neural Networks t -1 moment of hidden information, A dynamically deforming mesh representing the skeletal character at the target pose, Represents the skeleton structure of the skeleton character, represents the trainable skin weight matrix, Represents the LBS linear skinning function.

9. The modular integration method of 3D virtual morphing neural network based on Unreal Engine matching according to claim 1, characterized in that: The training of lightweight deformable neural network uses the following loss function: in, Represents a mesh node in a dynamically deforming mesh predicted by a lightweight deformable neural network The spatial position coordinates of Represents mesh nodes in the dynamic deformation mesh data of a flexible body The spatial position coordinates of Indicates the number of grid nodes, express Distance calculation method.

10. A terminal device comprising a processor, a memory, and a computer program stored in the memory; characterized in that: When the processor executes the computer program, the three-dimensional virtual deformation neural network modular integration method based on Unreal Engine matching is implemented.