Dressing data processing method and apparatus for virtual object, electronic device, computer readable storage medium, and computer program product

Abstract

The present application provides a dressing data processing method and apparatus for a virtual object, an electronic device, a computer readable storage medium, and a computer program product. The method comprises: acquiring object action data of a virtual object and model configuration information; acquiring a trained clothing prediction model on the basis of the model configuration information, wherein the trained clothing prediction model is obtained by training by using training action data of a reference virtual object and training point cloud data about a virtual clothing, and the training action data and the training point cloud data are obtained by performing offline cloth simulation on training animation data; using the clothing prediction model to perform prediction processing on the object action data to obtain vertex position information of each clothing vertex in the virtual clothing worn by the virtual object when executing the object action data; and on the basis of the vertex position information of each clothing vertex in the virtual clothing and attribute information of the virtual clothing, rendering the virtual clothing.

Description

Method, device, electronic device, computer-readable storage medium, and computer program product for processing virtual object clothing data

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the Chinese patent application with application number 202310936331.5 and application date July 28, 2023, and claims the priority of the above Chinese patent application. The entire content of the above Chinese patent application is hereby introduced into this application as a reference. Technical Field

[0003] The present application relates to data processing technology, and in particular to a method, device, electronic device, computer-readable storage medium, and computer program product for processing clothing data of a virtual object. Background Art

[0004] Cloth solving involves automatically adding clothing and fabric animation to a virtual object after the virtual object model has been created. Cloth solving can be applied to a variety of scenarios, including live streaming of virtual digital humans, offline video production, and gaming. Currently, mainstream cloth solving solutions are divided into two categories: offline and real-time. Offline cloth solving is typically performed in software such as Houdini or MD. This method requires high computing power, high labor costs, and is time-consuming. While solving time increases with the complexity of the garment, it provides a more precise solution and produces better results. Real-time cloth solving algorithms can generally only solve simple clothing models and suffer from poor results when solving complex clothing models.

[0005] Summary of the Invention

[0006] Embodiments of the present application provide a method, device, electronic device, computer-readable storage medium, and computer program product for processing clothing data of a virtual object, which can improve the prediction effect of clothing movement posture.

[0007] The technical solution of the embodiment of the present application is implemented as follows:

[0008] The present invention provides a method for processing clothing data of a virtual object, which is applied to an electronic device. The method includes:

[0009] Obtain object action data and model configuration information of the virtual object;

[0010] Obtaining a trained clothing prediction model based on the model configuration information, wherein the trained clothing prediction model is trained using training motion data of a reference virtual object and training point cloud data about the virtual clothing, wherein the training motion data and the training point cloud data are obtained by performing offline cloth solving on the training animation data;

[0011] Using the clothing prediction model to perform prediction processing on the object motion data, to obtain vertex position information of each clothing vertex in the virtual clothing when the virtual object wears the virtual clothing and performs the object motion data;

[0012] The virtual clothing worn by the virtual object when executing the action data is rendered based on the vertex position information of each clothing vertex in the virtual clothing and the attribute information of the virtual clothing.

[0013] An embodiment of the present application provides a device for processing clothing data of a virtual object, comprising:

[0014] a first acquisition module configured to acquire object motion data and model configuration information of a virtual object;

[0015] a second acquisition module configured to acquire a trained clothing prediction model based on the model configuration information, wherein the trained clothing prediction model is trained using training motion data of a reference virtual object and training point cloud data about a virtual garment, wherein the training motion data and the training point cloud data are obtained by performing offline cloth solving on the training animation data;

[0016] a prediction module configured to perform prediction processing on the object motion data using the clothing prediction model to obtain vertex position information of each clothing vertex in the virtual clothing when the virtual object wears the virtual clothing and performs the object motion data;

[0017] The rendering module is configured to render the virtual garment based on vertex position information of each garment vertex in the virtual garment and attribute information of the virtual garment.

[0018] An embodiment of the present application provides an electronic device, comprising:

[0019] a memory for storing computer-executable instructions;

[0020] The processor is configured to implement the virtual object clothing data processing method provided in an embodiment of the present application when executing the computer executable instructions stored in the memory.

[0021] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the virtual object clothing data processing method provided in the embodiment of the present application when executed by a processor.

[0022] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the method for processing clothing data of a virtual object provided in the embodiment of the present application is implemented.

[0023] The embodiments of the present application have the following beneficial effects:

[0024] After obtaining the object motion data and model configuration information of the virtual object, a trained clothing prediction model corresponding to the model configuration information is first obtained, wherein the trained clothing prediction model is trained using the training motion data of the reference virtual object and the training point cloud data of the virtual clothing. The training motion data and the training point cloud data are obtained by performing offline cloth solving on the training animation data. Since the offline cloth solving has a high degree of solution precision, it can ensure the accuracy of the training motion data and the training point cloud data, and further improve the prediction performance of the trained clothing prediction model. Therefore, when the trained clothing prediction model is used to predict the object motion data, the vertex position information of each clothing vertex in the virtual clothing worn by the virtual object performing the object motion data is obtained. Based on the vertex position information of each clothing vertex in the virtual clothing and the attribute information of the virtual clothing, when rendering the virtual clothing, it is possible to achieve high processing efficiency of real-time cloth solving while ensuring that the shape of the virtual clothing conforms to the movement of the virtual object, thereby improving the prediction effect of the motion posture of the virtual clothing and the authenticity of the animation playback. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] FIG1 is a schematic diagram of the network architecture of a live broadcast system 100 provided in an embodiment of the present application;

[0026] FIG2 is a schematic structural diagram of an anchor terminal 400 provided in an embodiment of the present application;

[0027] FIG3 is a schematic diagram of an implementation flow of a method for processing clothing data of a virtual object provided in an embodiment of the present application;

[0028] FIG4 is a schematic diagram of an implementation flow of determining vertex position information of each vertex of a virtual garment provided by an embodiment of the present application;

[0029] 5 is a schematic diagram of an implementation flow of determining vertex position information of each garment vertex in a virtual garment using a trained garment prediction model according to an embodiment of the present application;

[0030] FIG6 is a schematic diagram of an implementation process of rendering virtual clothing provided by an embodiment of the present application;

[0031] FIG7 is a schematic diagram of a process for implementing collision correction for clothing vertices of a virtual clothing according to an embodiment of the present application;

[0032] FIG8 is a schematic diagram of an implementation flow of a processor for determining and executing virtual clothing prediction according to an embodiment of the present application;

[0033] FIG9 is a schematic diagram of an implementation flow of a training clothing prediction model provided in an embodiment of the present application;

[0034] FIG10 is a schematic diagram of another implementation flow of a training clothing prediction model provided in an embodiment of the present application;

[0035] FIG11 is a schematic diagram of an implementation flow of training a low-frequency prediction module according to an embodiment of the present application;

[0036] FIG12 is a schematic diagram of an implementation flow of training a high-frequency prediction module according to an embodiment of the present application;

[0037] FIG13A is a schematic diagram of a frame of a single skirt rendered through offline computing;

[0038] FIG13B is a diagram showing the effect of real-time prediction of a skirt using the trained clothing prediction model provided in an embodiment of the present application;

[0039] FIG13C is a diagram showing the effect of real-time prediction in the UE engine using the trained clothing prediction model provided in an embodiment of the present application;

[0040] FIG13D is another effect diagram of real-time prediction in the UE engine using the trained clothing prediction model provided by an embodiment of the present application;

[0041] FIG14 is a schematic diagram of another implementation flow of the method for rendering virtual clothing provided in an embodiment of the present application;

[0042] FIG15 is a rendering of a garment at the 200-frame stage after a rotation animation is completed;

[0043] FIG16 is an architectural diagram of a UE engine according to an embodiment of the present application;

[0044] FIG17 is a schematic diagram of an implementation flow of cloth simulation on the engine side provided by an embodiment of the present application;

[0045] FIG18 is a schematic diagram of the cloth rendering process provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0047] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0048] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0049] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be fully or partially implemented by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0050] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0051] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0052] 1) Unreal Engine (UE): A commercial game engine used in the development of various game genres, including stealth games and role-playing games. In addition to game development, it is also used in computer virtual production and virtual human development.

[0053] 2) Cloth simulation: A method for performing physical simulation on a cloth model, which in the embodiment of the present application refers to simulating various states of cloth moving with the movement of a character.

[0054] 3) Deep Learning (DL), which is used to teach computers to process data in a way similar to how neurons in the human brain conduct information.

[0055] 4) Deep learning models: Models built using deep learning methods can process different data and be used to recognize images, text, sounds, or other complex patterns to complete specified tasks.

[0056] 5) Open Neural Network Exchange (ONNX), an open format proposed by Microsoft and Facebook for representing deep learning models, defines a set of standard formats that are independent of the environment and platform to enhance the interoperability of various AI models.

[0057] 6) Eigen is an open source template library in C++ language that supports linear algebra operations, matrix and vector operations, numerical analysis and related algorithms.

[0058] 7) Compute Unified Device Architecture (CUDA) is a parallel computing programming interface that allows programs to use certain types of graphics processing units (GPUs) for general processing.

[0059] 8) Houdini is a software for 3D animation developed by Side Effects Software Inc. of Canada.

[0060] 9) Animation retargeting, a method that allows animations to be reused between different characters with similar skeleton assets, allowing the same animation data to drive the movements of different characters.

[0061] 10) Recurrent neural network is a deep learning neural network that uses sequence data or time series data and is often used for sequential and time problems.

[0062] 11) Gated Recurrent Unit (GRU) is a type of recurrent neural network with a simple structure and easy training.

[0063] 12) Hidden layer: abstracts the features of the input data into another dimensional space to display more abstract features.

[0064] 13) Temporal Anti-Aliasing (TAA) is an anti-aliasing method, also known as an anti-spatial aliasing technology, and is a commonly used image enhancement algorithm.

[0065] 14) Temporal Super Resolution (TSR) is an anti-aliasing technology released on UE5 that scales low-resolution frames to normal size for display. It has high real-time and bandwidth overhead.

[0066] 15) Skin weight, which represents the degree of association between each skin point and the bone when animating a character. It determines how the character deforms and moves during the animation process.

[0067] 16) Cloth solving refers to the use of computer programs to simulate the elastic deformation and movement of cloth to reflect the real-world response of cloth to external forces. In games or simulations, cloth solving can be used to simulate and reproduce the movement and collision of soft objects such as clothing, curtains, and coverings, making the virtual environment more realistic and natural. Cloth solving typically requires the use of specific algorithms and models to accurately simulate the elasticity, friction, and dynamic deformation of cloth, thereby providing players or viewers with more realistic visual and physical feedback.

[0068] In order to better understand the virtual object clothing data processing method provided in the embodiment of the present application, the fabric solution algorithm used for virtual clothing prediction in the related art and its existing shortcomings are first explained.

[0069] The fabric solving schemes currently used for virtual clothing prediction are divided into two categories: offline fabric solving and real-time fabric solving. Offline fabric solving is generally performed in Houdini and MD software, which has high processor occupancy, high labor costs, and is time-consuming. The solving time increases with the complexity of the clothing, but the solving is precise and the effect is better.

[0070] Most existing real-time cloth solving methods are based on physics, which requires large amounts of data and consumes a lot of hardware resources. Currently, there are mainly the following types of real-time cloth solving methods:

[0071] 1. UE's built-in cloth solving system is a physical solving method.

[0072] 2. Udrapper cloth solving, which is a physical solution, uses GPU acceleration, and the effect is between offline and real-time solving.

[0073] 3. UE ML Deformed is UE's built-in deep learning-based solution method, which is often used to simulate muscles and can also be applied to clothing. It only requires 1,000 frames of offline solution data for training.

[0074] Several real-time cloth solving methods in related technologies have the following shortcomings:

[0075] 1. The UE's built-in fabric solving system can only solve relatively simple effects and relatively simple models, such as ribbons, flags, and complex models (including single-layer complex models and multi-layer models). The fabric details are insufficient and the effect is unstable during live broadcast.

[0076] 2. Udrapper fabric solver can achieve real-time solving for simple clothing models, but it cannot support general models. It requires the use of Udrapper's proprietary model format and has limitations on clothing model accuracy. Complex models (including single-layer complex models and multi-layer models) may have problems with threading and cannot be solved in real time.

[0077] 3. UE ML Deformed generally only produces wrinkles for tight fabrics and is used to refine the calculation effect, but it cannot be used for loose fabrics and has no inertial dynamics.

[0078] 4. Based on the physics-based solution, the more complex the clothing is, the more computing power is consumed, resulting in low clothing rendering efficiency.

[0079] Based on this, the embodiments of the present application provide a method, apparatus, device, computer-readable storage medium, and computer program product for processing the clothing data of a virtual object, which can improve the prediction effect of the motion posture of the virtual clothing and the authenticity of the animation playback when performing real-time solution of the clothing of the virtual object. The following describes an exemplary application of the electronic device provided by the embodiment of the present application. The electronic device provided by the embodiment of the present application can be implemented as various types of user terminals such as laptops, tablet computers, desktop computers, set-top boxes, mobile devices (for example, mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), smart phones, smart speakers, smart watches, smart TVs, and vehicle-mounted terminals, and can also be implemented as servers. Below, an exemplary application when the device is implemented as a terminal will be described.

[0080] Refer to Figure 1, which is a schematic diagram of the network architecture of the live broadcast system 100 provided in an embodiment of the present application. As shown in Figure 1, the network architecture includes a server 200, a network 300, an anchor terminal 400 and an audience terminal 500, wherein the anchor terminal 400 and the audience terminal 500 are respectively connected to the server 200 through the network 300, and the network 300 can be a wide area network or a local area network, or a combination of the two.

[0081] The host terminal 400 can schedule a live broadcast through a live broadcast app, instant messaging app, shopping app, etc., and after the live broadcast reservation is completed, the live broadcast server will assign a live broadcast identifier and the push stream address and pull stream address corresponding to the live broadcast identifier to the live broadcast. Before the live broadcast begins, the host terminal 400 can obtain configuration information for the virtual digital human live broadcast, where the configuration information includes the virtual digital human identifier for this live broadcast and the clothing identifier of the clothing worn by the virtual digital human during the live broadcast. The host terminal 400 determines the trained clothing prediction model for virtual clothing prediction based on the clothing identifier. At the beginning of the live broadcast, the game engine in the host terminal 400 receives the live animation data collected in real time by the motion capture device, and then calls the animation blueprint in the game engine to obtain the motion data. Using the trained clothing prediction model, motion data, and virtual clothing attribute information, it determines the movement posture of the virtual digital human's virtual clothing when executing the motion data. The trained clothing prediction model is trained using the training motion data of the virtual object and the training point cloud data of the virtual clothing. The training motion data and the training point cloud data are obtained by performing offline cloth solving on the training animation data. Since the training motion data and the training point cloud data are obtained by offline cloth solving, the accuracy of the position information of each clothing vertex predicted by the trained clothing prediction model can be guaranteed, thereby improving the authenticity of the live broadcast of the virtual digital human. The anchor terminal 400 sends the live data stream of the virtual object wearing the virtual clothing and performing the action to the server 200. The server 200 sends the live data to the audience terminal 500 through the network 300 to present the live data on the display screen of the audience terminal 500.

[0082] In some embodiments, the server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The anchor terminal 400 and the audience terminal 500 can be smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, car terminals, etc., but are not limited to these. The terminals and servers can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0083] Referring to Figure 2 , which is a schematic diagram of the structure of a host terminal 400 provided in an embodiment of the present application, the host terminal 400 shown in Figure 2 includes: at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the host terminal 400 are coupled together via a bus system 440. It will be understood that bus system 440 is used to enable connectivity and communication between these components. In addition to a data bus, bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in Figure 2, all of these buses are labeled as bus system 440.

[0084] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0085] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0086] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0087] The memory 450 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0088] In some embodiments, the memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0089] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0090] A network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include Bluetooth, Wi-Fi, and Universal Serial Bus (USB);

[0091] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0092] The input processing module 454 is configured to detect one or more user inputs or interactions from one of the one or more input devices 432 and to translate the detected inputs or interactions.

[0093] In some embodiments, the apparatus provided by the embodiments of the present application can be implemented using software. FIG2 shows a virtual object clothing data processing device 455 stored in memory 450. This device 455 can be software in the form of a program or plug-in, and includes the following software modules: a first acquisition module 4551, a second acquisition module 4552, a prediction module 4553, and a rendering module 4554. These modules are logical and can be arbitrarily combined or further separated according to the functions implemented. The functions of each module will be described below.

[0094] In other embodiments, the apparatus provided in the embodiments of the present application may be implemented in hardware. As an example, the apparatus provided in the embodiments of the present application may be a processor in the form of a hardware decoding processor, which is programmed to execute the virtual object clothing data processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor may be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0095] In order to better understand the method for processing the clothing data of a virtual object provided in the embodiment of the present application, artificial intelligence, its various branches, and the application fields involved in the method for processing the clothing data of a virtual object provided in the embodiment of the present application are first explained.

[0096] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0097] Artificial intelligence technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Basic AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning. The methods for processing virtual object clothing data provided in the present embodiments primarily address machine learning, which are described separately below.

[0098] Key technologies in speech technology include Automatic Speech Recognition (ASR), Text-to-Speech (TTS), and voiceprint recognition. Enabling computers to hear, see, speak, and feel is the future direction of human-computer interaction, with speech becoming one of the most promising methods of human-computer interaction.

[0099] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning.

[0100] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0101] The method for processing clothing data of a virtual object provided by an embodiment of the present application will be described in conjunction with the exemplary application and implementation of the terminal provided by the embodiment of the present application.

[0102] The following describes a method for processing virtual object clothing data provided by an embodiment of the present application. As previously mentioned, the electronic device implementing the method for processing virtual object clothing data in the embodiment of the present application can be a terminal, a server, or a combination of the two. Therefore, the following description of the execution entity of each step is omitted.

[0103] Refer to Figure 3, which is a schematic diagram of an implementation flow of a method for processing clothing data of a virtual object provided in an embodiment of the present application. The steps shown in Figure 3 will be explained. The main body of the steps in Figure 3 is the terminal.

[0104] In step 101 , object motion data and model configuration information of a virtual object are obtained.

[0105] In some embodiments, the object motion data may be obtained from live animation data collected in real time or from pre-generated animation data. The object motion data includes the rotation angles of the key bones of the virtual object and the displacement of the root bone in each animation frame in the animation data.

[0106] When implementing step 101, first obtain the animation data to be processed, where the animation data is live animation data collected in real time. The live animation data can be collected in real time using a motion capture device, or the animation data can be pre-generated animation data; then use the game engine to process the animation data to obtain the rotation angle of the key bones preset in each animation frame of the virtual object and the displacement of the root bone of the virtual object.

[0107] In some embodiments, the game engine can be a UE engine, a Unity engine, or other types of game engines. After the animation data to be processed is imported into the game engine, the animation blueprint can be called to obtain the rotation angle of the key bones preset in each animation frame of the virtual object and the displacement of the root bone of the virtual object. Among them, the key bones of the virtual object are pre-screened from the multiple bones that constitute the virtual object, and are bones whose positions change to a large extent during the movement of the virtual object. Exemplarily, a virtual object can have 22 key bones.

[0108] In some embodiments, the model configuration information includes: the type of clothing prediction model, the clothing identifier of the virtual clothing, and the object identifier of the virtual object.

[0109] In step 102, a trained clothing prediction model is obtained based on the model configuration information.

[0110] In some embodiments, when a virtual object wears virtual clothing of different styles or materials, even when performing the same action, the movement amplitude and deformation of the virtual clothing may be different. For example, when a virtual object performs a circle action while wearing a virtual clothing made of denim material, the movement amplitude of the virtual clothing is very different from when the virtual object performs a circle action while wearing a virtual clothing made of silk material. Therefore, to ensure the accuracy of virtual clothing prediction, animation data of virtual objects wearing virtual clothing of the same material and style can be used as training data to train the clothing prediction model to obtain a trained clothing prediction model. In other words, different virtual clothing corresponds to different trained clothing prediction models. In addition, to meet different rendering quality requirements, animation data of virtual clothing of the same material and style can also be used to train clothing prediction models of different types or different model structures.

[0111] Based on this, when step 102 is implemented, the trained clothing prediction model is obtained based on the type of clothing prediction model in the model configuration information and the clothing identifier of the virtual clothing.

[0112] The trained clothing prediction model is trained using training motion data of a reference virtual object and training point cloud data of a virtual garment. The reference virtual object can be the same object as the virtual object in step 101 or a different object. The training motion data and training point cloud data are obtained by performing offline fabric solution on the training animation data. Because the training motion data and training point cloud data obtained by offline fabric solution retain intact motion and garment details, the accuracy of the vertex position information of garment vertices can be guaranteed when the trained clothing prediction model is used to predict the motion data.

[0113] In step 103, the object action data is predicted and processed using the clothing prediction model to obtain vertex position information of each clothing vertex in the virtual clothing when the virtual object wears the virtual clothing and performs the object action data.

[0114] In some embodiments, referring to FIG. 4 , step 103 may be implemented through steps 1031 to 1033 , which are described in detail below.

[0115] In step 1031, the rotation angles of the key bones and the displacements of the root bones of the virtual object in each animation frame are normalized respectively to obtain the normalized rotation angles of the key bones and the normalized displacements of the root bones of the virtual object in each animation frame.

[0116] In some embodiments, when normalizing the rotation angle of the key bone, the rotation angle of the key bone is first converted into rotation radians, and the value range of the rotation radians is [-π, π]. Then, the rotation radians of the key bone are converted to the value range of [0, 1] to obtain the normalized rotation angle of the key bone.

[0117] In some embodiments, when normalizing the displacement of the root bone, the mean and variance of the root bone displacement can be obtained first, and then the displacement of the root bone can be normalized according to the method of (displacement of the root bone - mean) / variance to obtain the normalized displacement of the root bone.

[0118] In step 1032, the velocity of the root bone of the virtual object in each animation frame is determined based on the normalized displacement of the root bone of the virtual object in each animation frame.

[0119] In some embodiments, the normalized displacement of the root bone of the virtual object in the previous animation frame of the current animation frame is first subtracted from the normalized displacement of the root bone of the virtual object in the current animation frame to obtain the displacement difference, and then the displacement difference is divided by the time difference between the two animation frames to obtain the speed of the root bone of the virtual object in the current animation frame.

[0120] In step 1033, the trained clothing prediction model is used to predict the normalized rotation angles of the key bones and the speed of the root bones of the virtual object in each animation frame to obtain the vertex position information of each clothing vertex in the virtual clothing when the virtual object wears the virtual clothing and performs the object action data.

[0121] In some embodiments, the trained clothing prediction model includes at least a trained low-frequency prediction module and a trained high-frequency prediction module. Step 1033 can be implemented through steps 331 to 334 shown in FIG5 , which will be described in detail below.

[0122] In step 331, the trained low-frequency prediction module is used to predict the normalized rotation angle of the key bones and the speed of the root bones of the virtual object in each animation frame to obtain the first predicted rotation angle and first predicted position of the virtual bones of the virtual clothing in each animation frame.

[0123] In some embodiments, the low-frequency prediction module can be a recurrent neural network model with a GRU as the recurrent unit. When the trained low-frequency prediction module is used to predict the normalized rotation angle of the key bones and the speed of the root bones of the virtual object in each animation frame, the input data of the trained low-frequency prediction module includes two parts, one part is the hidden layer data of the previous frame, and the other part is the normalized rotation angle of the key bones and the speed of the root bones of the virtual object in the current animation frame. The output is the first predicted rotation angle and the first predicted position of the virtual bones of the virtual object in the current animation frame. For the first animation frame, the hidden layer data of the previous frame is a preset value.

[0124] In step 332 , the first reference position of each garment vertex of the virtual garment in each animation frame is determined according to the first predicted rotation angle, the first predicted position and the skin weight of the virtual skeleton of the virtual garment in each animation frame.

[0125] In some embodiments, the first predicted rotation angle, first predicted position and skin weight of the virtual skeleton of the virtual garment in each animation frame can be input into a position-based service model to determine the first reference position of each garment vertex of the virtual garment in each animation frame. In actual applications, each garment vertex can be associated with one or more bones, and the skin weight defines the weight value of these associations. The larger the weight value, the higher the degree of association between the garment vertex and the bone, and the more significant the deformation effect. This technology is widely used in computer games, animated films and other fields. Skin weights are usually obtained through three-dimensional animation production software (such as Maya, 3D Max, etc.). For example, in Maya, after completing the selection of the model and the root node of the skeleton, binding is then performed through the menu Skin>Bind Skin. After binding is completed, skin weights are automatically assigned to each vertex. In some embodiments, these weight values ​​can also be manually adjusted as needed.

[0126] When determining the first reference position of each garment vertex of the virtual garment in each animation frame, the position-based service model first determines the rotation matrix and translation matrix of the virtual garment in each animation frame based on the first predicted rotation angle and first predicted position of the virtual skeleton of the virtual garment in each animation frame, and then determines the first reference position of each garment vertex of the virtual garment in each animation frame by solving the skinning calculation formula.

[0127] In step 333, the trained high-frequency prediction module is used to predict the normalized rotation angle of the key bones and the speed of the root bone of the virtual object in each animation frame, and the vertex offset information of each clothing vertex of the virtual clothing in each animation frame is obtained.

[0128] In some embodiments, the high-frequency prediction module can be a recurrent neural network model with a GRU as the recurrent unit. The input data of the trained high-frequency prediction module is similar to that of the low-frequency prediction module, consisting of two parts: one part is the hidden layer data of the previous frame, where, for the first animation frame, the hidden layer data of the previous frame is a preset value. The other part is the normalized rotation angle of the key bones of the virtual object in the current animation frame and the velocity of the root bone. The output is the vertex offset information of each garment vertex in each animation frame.

[0129] In step 334 , the corresponding vertex offset information is superimposed on the first reference position of each vertex of the virtual garment in each animation frame to obtain vertex position information of each vertex of the virtual garment in each animation frame.

[0130] In some embodiments, the first reference position of the clothing vertex includes the x reference coordinate and the y reference coordinate of the vertex; the vertex offset information includes the x coordinate offset and the y coordinate offset. When step 334 is implemented, the x reference coordinate of the vertex is added to the x coordinate offset to obtain the final predicted x coordinate; similarly, the y reference coordinate of the vertex is added to the y coordinate offset to obtain the final predicted y coordinate.

[0131] In the above steps 1031 to 1033, the trained low-frequency prediction module is used to roughly predict the position of each virtual bone in the virtual clothing, and then the position of each virtual bone in the virtual clothing is converted into the rough vertex position information of each clothing vertex in the virtual clothing, and then the high-frequency prediction module is used to predict the precise vertex position of each clothing vertex of the virtual clothing and the offset of the rough vertex position information, so as to determine the final vertex position information of each service vertex based on the rough vertex position information and offset of each clothing vertex. Compared with the related technology, by performing weighted summation on the action network under different parameters, it can reduce hardware consumption, and has high processing efficiency, which can meet strong real-time requirements.

[0132] 3 , the following description continues with step 103 .

[0133] In step 104 , the virtual garment is rendered based on the vertex position information of each garment vertex in the virtual garment and the attribute information of the virtual garment.

[0134] In some embodiments, the attribute information of the virtual clothing includes at least the texture coordinates of each clothing vertex. Referring to FIG. 6 , step 104 can be implemented through the following steps 1041 to 1044 , which are described in detail below.

[0135] In step 1041 , a capsule model corresponding to the virtual object is obtained.

[0136] In some embodiments, the two arms, two legs, torso, and head of a virtual object are each considered a capsule, that is, the corresponding capsule model of the virtual object is considered a combination of multiple capsules. Each capsule can be considered a combination of two hemispheres and a cylinder.

[0137] In step 1042 , based on the vertex position information of each garment vertex in the virtual garment and the capsule model, collision detection and correction are performed on each garment vertex in the virtual garment to obtain corrected vertex position information of each garment vertex.

[0138] In some embodiments, referring to FIG. 7 , step 1042 may be implemented through steps 421 to 424 , which are described in detail below.

[0139] In step 421 , a center line of the capsule model is determined, and a vertical distance between each garment vertex and the center line is determined based on vertex position information of each garment vertex.

[0140] In some embodiments, determining the midline of the capsule model involves determining the midline of each capsule comprising the capsule model. The midline of each capsule can be a line connecting the centers of the hemispheres at either end of the capsule and extending outward to a vertex on the capsule. Once the coordinates of the centers of the two hemispheres at either end of the capsule are known, a function expression for a line passing through the two centers can be determined. Thus, given the function expression for a line and the position information of the garment vertices, the perpendicular distance between each garment vertex and the midline can be determined.

[0141] In step 422 , the garment vertices whose vertical distances are smaller than the reference distance are determined as vertices to be corrected that collide.

[0142] The reference distance is the distance from the intersection of the perpendicular line from the garment vertex to the capsule's midline and the capsule's midline. When the garment vertex is outside the capsule, meaning it has not collided with the virtual object, the intersection is between the garment vertex and the capsule's midline, and the reference distance is greater than the perpendicular distance. When the garment vertex is inside the capsule, meaning it has collided with the virtual object, the distance from the intersection to the capsule's midline is greater than the distance from the garment vertex to the capsule's midline. Therefore, in this step, garment vertices with perpendicular distances less than the reference distance are identified as vertices that have collided and need to be corrected.

[0143] In step 423, the difference between the reference distance and the vertical distance is determined as the offset distance of the vertex to be corrected.

[0144] In some embodiments, since the reference distance corresponding to the vertex to be corrected is greater than the vertical distance, the vertical distance is subtracted from the reference distance to obtain the offset distance of the vertex to be corrected.

[0145] In step 424 , the vertex position information of the vertex to be corrected is moved by the offset distance in the opposite direction of the perpendicular line from the vertex to be corrected to the midline to obtain the corrected vertex position information of the vertex to be corrected.

[0146] In some embodiments, given the known function expression of the capsule midline and the vertex position information of the vertex to be corrected, a function expression of the perpendicular line from the vertex to be corrected to the midline can be determined, and then the slope of the perpendicular line can be determined based on the function expression of the perpendicular line. Then, a first offset value of the vertex to be corrected in the x-direction and a second offset value in the y-direction can be determined by the offset distance and the slope of the perpendicular line. Finally, the x-coordinate of the vertex to be corrected is added to the first offset value to obtain the corrected x-coordinate of the vertex to be corrected. Similarly, the y-coordinate of the vertex to be corrected is added to the second offset value to obtain the corrected y-coordinate of the vertex to be corrected.

[0147] In the above steps 421 to 424, the capsule model is used to perform collision detection on each clothing vertex in the virtual clothing. Since the capsule model can better capture and represent the shape and structural information of the virtual clothing, including complex clothing details and deformation, potential collisions between clothing vertices can be detected more accurately. Moreover, through the collision detection correction of the capsule model, the vertex position of the virtual clothing can be automatically adjusted to prevent part of the body area of ​​the virtual object from penetrating the virtual clothing, thereby improving the authenticity of clothing wearing while ensuring that the virtual complexities better fit the virtual object.

[0148] 6 , and proceed to step 1042 for further description.

[0149] In step 1043, the vertex normals of the respective clothing vertices are obtained.

[0150] The normal of a three-dimensional plane is a three-dimensional vector perpendicular to the plane, and the normal of a surface at a certain point P is a vector perpendicular to the tangent plane of the point. In some embodiments, when step 1043 is implemented, the triangular facets where each clothing vertex is located are first determined; when the clothing vertex exists in only one triangular facet, the vertex normal of the clothing vertex is determined based on the corrected vertex position information of the three clothing vertices of the triangular facet; when the clothing vertex exists in at least two triangular facets, then the clothing vertex has a vertex normal in each triangular facet, and the vertex normal of the clothing vertex in each triangular facet is determined at this time; and the vertex normal of the clothing vertex in each triangular facet is weightedly calculated to obtain the vertex normal of the clothing vertex.

[0151] In some embodiments, when performing weighted calculation on the vertex normals of each triangle facet of a clothing vertex, the weights of each triangle facet can be set to be the same, that is, the vertex normals of at least two triangle facets of the clothing vertex are averaged to obtain the final vertex normal of the clothing vertex; in addition, the corresponding weight can be determined based on the area of ​​each triangle facet, and then the weighted calculation of at least two triangle facets of the clothing vertex is performed according to the corresponding weight of each triangle facet to obtain the final vertex normal of the clothing vertex.

[0152] In some embodiments, when determining the corresponding weight based on the area of ​​each triangle, the area of ​​each triangle is accumulated to obtain the total area, and then the ratio of the area of ​​each triangle to the total area is determined as the weight corresponding to each triangle. Since the larger the area of ​​the triangle, the greater the proportion of the triangle in the virtual clothing, and the greater the impact on the normal, giving a larger weight to the triangle with a larger area can improve the accuracy of the vertex normal of the clothing vertex.

[0153] In step 1044 , the virtual clothing that the virtual object wears when executing the action data is rendered based on the corrected vertex position information, vertex normals, and texture coordinates of each clothing vertex.

[0154] In some embodiments, the attribute information of the virtual clothing also includes material information, color information or mapping information, and then based on the corrected position information of the clothing vertices, vertex normals, texture coordinates and material information, color information, and mapping data, a rendering tool is used to render the virtual clothing worn by the virtual object when performing the action data.

[0155] In the above steps 1041 to 1044, when rendering the virtual clothing worn by the virtual object when executing the action data based on the vertex position information of each clothing vertex in the virtual clothing and the attribute information of the virtual clothing, a collision correction detection will be performed on each clothing vertex on the virtual clothing, so that the clothing vertices located inside the virtual object are moved to the outside of the virtual object to achieve position correction of the clothing vertices, and then the normals of each clothing vertex are obtained. Finally, based on the corrected vertex position information, vertex normals and texture coordinates of each clothing vertex, the virtual clothing worn by the virtual object when executing the action data is rendered; and if the first number of the texture coordinate is greater than the second number of the vertex position information of each clothing vertex, the vertex position information of the second number can be mapped to the vertex position information of the first number through a pre-calculated mapping table to achieve rendering data supplementation, ensure the consistency of the number of texture coordinates and the number of vertex position information, thereby improving the accuracy and authenticity of the rendering results.

[0156] After obtaining the object motion data and model configuration information of the virtual object, the trained clothing prediction model corresponding to the model configuration information is first obtained, wherein the trained clothing prediction model is trained using the training motion data and the training point cloud data about the virtual clothing, and the training motion data and the training point cloud data are obtained by performing offline cloth solution on the training animation data. Since the training motion data and the training point cloud data are obtained by offline cloth solution, the training motion data and the training point cloud data include more refined motion details and clothing details, so that the motion data is predicted and processed using the trained clothing prediction model to obtain the vertex position information of each clothing vertex in the virtual clothing, and based on the vertex position information of each clothing vertex in the virtual clothing and the attribute information of the virtual clothing, when the virtual clothing is rendered, the movement of the virtual clothing can be made to conform to the movement of the virtual object, thereby improving the rendering effect of the virtual clothing and the authenticity of the animation playback.

[0157] In some embodiments, when the trained clothing prediction model is used to predict the action data and obtain the virtual object wearing virtual clothing to perform object action data, before the vertex position information of each clothing vertex in the virtual clothing is obtained, the processor selection can also be performed through steps 201 to 209 shown in Figure 8, which is explained in detail below.

[0158] In step 201, the current CPU usage and GPU usage are obtained.

[0159] In some embodiments, the CPU usage is a real number between 0 and 1, for example, the CPU usage is 20%. Similarly, the GPU usage is also a real number between 0 and 1, for example, the GPU usage is 40%.

[0160] In step 202 , it is determined whether the CPU usage is less than a first usage threshold, and whether the GPU usage is less than a second usage threshold.

[0161] The first usage threshold and the second usage threshold are pre-set and can be the same or different. For example, the first usage threshold is 50% and the second usage threshold is 60%. If the CPU usage is less than the first usage threshold and the GPU usage is less than the second usage threshold, proceed to step 203. If the CPU usage is not less than the first usage threshold and the GPU usage is not less than the second usage threshold, proceed to step 205.

[0162] In step 203, processor selection information is obtained.

[0163] In some embodiments, if the CPU usage is less than a first usage threshold and the GPU usage is less than a second usage threshold, indicating that neither the CPU nor the GPU is in a high-load operating state, processor selection information can be obtained, wherein the processor selection information includes the processor currently used for virtual clothing prediction processing. The processor selection information has a default value, such as the default value of the CPU. The processor selection information can also be determined based on a user's selection operation for the processor. For example, the user can set the GPU to perform virtual clothing rendering when neither the CPU nor the GPU is in a high-load operating state. In this case, the processor selection information is the GPU.

[0164] In step 204, the processor corresponding to the processor selection information is determined as the target processor.

[0165] Among them, the target processor is used to execute prediction processing on the action data using the trained clothing prediction model, obtain vertex position information of each clothing vertex in the virtual clothing, and based on the vertex position information of each clothing vertex in the virtual clothing and the attribute information of the virtual clothing, render the virtual clothing worn by the virtual object when executing the action data.

[0166] In step 205 , it is determined whether the GPU usage rate is less than a second usage rate threshold.

[0167] Since step 205 is entered under the premise that the CPU usage is not less than the first usage threshold and the GPU usage is not less than the second usage threshold, then if the GPU usage is less than the second usage threshold, it means that the CPU usage is greater than or equal to the first usage threshold, and step 206 is entered at this time; if the GPU usage is greater than or equal to the second usage threshold, it cannot be determined whether the CPU usage is less than the first usage threshold, and step 207 is entered at this time.

[0168] In step 206 , the GPU is determined as the target processor.

[0169] In some embodiments, when the GPU usage is less than the second usage threshold and the CPU usage is greater than or equal to the first usage threshold, it means that the CPU is in a high-load working state and the GPU is not in a high-load working state. At this time, the GPU is determined as the target processor, and the rendering process of the virtual clothing is implemented through the GPU.

[0170] In step 207, it is determined whether the CPU usage is less than a first usage threshold.

[0171] If the CPU usage is less than the first usage threshold, the process proceeds to step 208 ; if the CPU usage is greater than or equal to the first usage threshold, the process proceeds to step 209 .

[0172] In step 208 , the CPU is determined as the target processor.

[0173] Since step 208 is performed under the premise that the CPU usage is less than the first usage threshold and the GPU usage is greater than or equal to the second usage threshold, that is, the GPU is in a high-load working state and the CPU is not in a high-load working state, the rendering process of the virtual clothing is realized by the CPU.

[0174] In step 209 , a processor with a low utilization rate among the GPU and the CPU is determined as a target processor.

[0175] In some embodiments, step 209 is performed under the premise that the CPU usage is greater than or equal to the first usage threshold and the GPU usage is greater than or equal to the second usage threshold, that is, the GPU is in a high-load working state and the CPU is also in a high-load working state. Then in step 209, the processor with low usage among the GPU and CPU is used to execute the rendering process of the virtual clothing.

[0176] In the above steps 201 to 209, the target processor for executing the rendering process of the virtual clothing can be determined based on the CPU usage and the GPU usage, thereby dynamically adjusting the use of the CPU or GPU based on the load of the hardware device, thereby ensuring the efficiency of the rendering process of the virtual clothing.

[0177] In some embodiments, before executing the above-mentioned virtual object clothing data processing method, the clothing prediction model to be trained can also be trained through steps 301 to 305 shown in Figure 9 to obtain a trained clothing prediction model, which is explained below in conjunction with Figure 9.

[0178] In step 301, training animation data and a clothing prediction model to be trained are obtained.

[0179] In some embodiments, the clothing prediction model may be a deep learning neural network model, such as a recurrent neural network model, and may include at least a low-frequency prediction module and a high-frequency prediction module. The low-frequency prediction module is used for rough prediction, and the high-frequency prediction module is used for precise prediction.

[0180] The training animation data can be animation data collected from real people performing common actions, or it can be animation data obtained by motion capturing a segment of live broadcast data and then redirecting the captured motion data to a reference virtual object.

[0181] In step 302, offline cloth solving is performed on the training animation data to obtain training motion data and training point cloud data of the virtual garment.

[0182] In some embodiments, the training animation data is input into an offline cloth solving tool to perform offline cloth solving on the training animation data, thereby obtaining training motion data and training point cloud data for the virtual garment. The training motion data includes the training rotation angles of the key bones of the reference virtual object and the training displacements of the root bones in each training animation frame. The training point cloud data for the virtual garment includes the vertex positions and vertex normals of each garment vertex during the execution of a motion by the reference virtual object wearing the virtual garment.

[0183] In some embodiments, offline cloth solving can be performed on the training animation data to generate training point cloud data for the virtual object, including information about multiple triangles in the virtual object and multiple triangles in the virtual garment. The training point cloud data for the reference virtual object includes the vertex positions and vertex normals of each vertex of the virtual object during the execution of the action.

[0184] In step 303 , the training rotation angles of the key bones and the training displacements of the root bones of the reference virtual object in each training animation frame are extracted from the training motion data.

[0185] In some embodiments, the training motion data is input into 3D animation software (eg, Maya) to extract the training rotation angles of key bones and the training displacements of the root bones of the reference virtual object in each training animation frame.

[0186] In step 304 , vertex label positions of the virtual garment vertices in each training animation frame are extracted from the training point cloud data of the virtual garment.

[0187] In step 305, the clothing prediction model to be trained is trained using the training rotation angles of the key bones and the training displacements of the root bones of the virtual object in each training animation frame, as well as the vertex label positions of the clothing vertices of the virtual clothing in each training animation frame, to obtain a trained clothing prediction model.

[0188] In some embodiments, referring to FIG. 10 , step 305 may be implemented through steps 3051 to 3053 , which are described in detail below.

[0189] In step 3051, the low-frequency prediction module to be trained is trained using the training rotation angles of the key bones and the training displacements of the root bones of the reference virtual object in each training animation frame, as well as the vertex label positions of the clothing vertices of the virtual clothing in the said each training animation frame, to obtain a trained low-frequency prediction module.

[0190] In some embodiments, referring to FIG. 11 , step 3051 may be implemented through steps 511 to 515 , which will be described below in conjunction with FIG. 11 .

[0191] In step 511 , based on the vertex label positions of the virtual garment vertices in each training animation frame, the bone label rotation angle, bone label position and skin weight of the virtual skeleton of the virtual garment in each training animation frame are determined.

[0192] In some embodiments, Laplace smoothing is first performed on the vertex label position of each garment vertex in each training animation frame to obtain the smoothed vertex label position of each garment vertex, and then skin decomposition is performed based on the smoothed vertex label position of each garment vertex to obtain a decomposition result, which is then imported into Maya to extract the bone label rotation angle, bone label position and skin weight of the virtual skeleton of the virtual garment in each training animation frame.

[0193] In step 512 , a plurality of training animation frames in the training animation data are divided into a plurality of training animation segments.

[0194] Wherein, two adjacent training animation segments include a preset number of overlapping training animation frames. For example, assuming that each training animation segment has 200 training animation frames, the 1st to 200th training animation frames are determined as the first training animation segment, the 101st to 300th training animation frames are determined as the second training animation segment, the 201st to 400th training animation frames are determined as the third training animation segment, and so on.

[0195] In some embodiments, if the reference virtual object's motion amplitude from the last frame of the previous training animation segment to the first frame of the next training animation segment is large, the virtual clothing's actual motion amplitude should also be large. When there are no overlapping training animation frames in adjacent training animation segments, the accumulated hidden layer weights of the previous training animation segment will be lost when predicting the first frame of the next training animation segment. Based on this, the trained clothing prediction model inputs the training data of each animation frame frame by frame when predicting. When predicting the first frame of the next animation segment, the accumulated hidden layer weights of multiple animation frames of the previous training animation segment will be added. The network will learn the inertia of the movement, resulting in the skirt of the first frame of the next animation segment, which should have been extremely large, not being able to swing to such a large amplitude due to the limitation of the hidden layer accumulation, thereby reducing the prediction accuracy. In the embodiment of the present application, the two adjacent training animation segments are divided into multiple overlapping training animation frames. In this way, it can ensure that the trained clothing prediction model can distinguish between motion inertia and the presence of large amplitude movement, thereby improving the prediction accuracy.

[0196] In step 513, the training rotation angle of the key bones and the training displacement of the root bones of the reference virtual object in each training animation frame of each training animation segment are predicted using the low-frequency prediction module to obtain the second predicted rotation angle and second predicted position of the virtual bones of the virtual clothing in each training animation frame.

[0197] In some embodiments, it is first necessary to determine the training speed of the root bone of the reference virtual object in each training animation frame based on the training displacement of the root bone of the reference virtual object in each training animation frame of each training animation segment, and then input the training rotation angle of the key bone and the training speed of the root bone of the reference virtual object in each training animation frame into the low-frequency prediction module frame by frame, so as to obtain the second predicted rotation angle and second predicted position of the virtual bone of the virtual clothing in each training animation frame.

[0198] In step 514 , a third reference position of each garment vertex of the virtual garment in each training animation frame is determined based on the second predicted rotation angle, second predicted position and skin weight of the virtual skeleton of the virtual garment in each training animation frame.

[0199] The implementation process of step 514 is similar to the implementation process of step 332 , and reference may be made to the implementation process of step 332 .

[0200] In step 515 , the low-frequency prediction module to be trained is trained using the third reference position and vertex label position of each garment vertex of the virtual garment in each training animation frame to obtain a trained low-frequency prediction module.

[0201] In some embodiments, a first difference between a third reference position and a vertex label position of each garment vertex of the virtual garment in each training animation frame is determined, and then the third reference position and the vertex label position of each garment vertex of the virtual garment in each training animation frame are smoothed respectively to obtain a smoothed third reference position and a smoothed vertex label position, and then a second difference between the smoothed third reference position and the smoothed vertex label position of each garment vertex of the virtual garment in each training animation frame is determined, and the sum of the first difference and the second difference is determined as a loss value of the low-frequency prediction module, and the loss value is back-propagated to the low-frequency prediction module, and the parameters of the low-frequency prediction model are adjusted using the gradient descent method until the training end condition is met to obtain a trained low-frequency prediction module.

[0202] In step 3052, the second reference position of each clothing vertex of the virtual clothing in each training animation frame is determined by using the trained low-frequency prediction module and the training rotation angles of the key bones and the training displacements of the root bones of the reference virtual object in each training animation frame of each training animation segment.

[0203] In some embodiments, after completing the training of the low-frequency prediction module, the trained low-frequency prediction module is used to determine the second reference position of each clothing vertex of the virtual clothing in each training animation frame. During implementation, based on the training displacement of the root bone of the reference virtual object in each training animation frame of each training animation segment, the training speed of the root bone of the reference virtual object in each training animation frame is determined, and then the training rotation angle and training speed of the key bones of the reference virtual object in each training animation frame are input into the trained low-frequency prediction module frame by frame, thereby obtaining the third predicted rotation angle and third predicted position of the virtual bones of the virtual clothing in each training animation frame, and then using the third predicted rotation angle and third predicted position of the virtual bones of the virtual clothing in each training animation frame and the skin weight, the second reference position of each clothing vertex of the virtual clothing in each training animation frame is determined.

[0204] In step 3053, the high-frequency prediction module to be trained is trained using the training rotation angles of the key bones and the training displacements of the root bones of the reference virtual object in each training animation frame of each training animation segment, the vertex label positions of the clothing vertices of the virtual clothing in the said each training animation frame and the second reference position to obtain a trained high-frequency prediction module.

[0205] In some embodiments, referring to FIG. 12 , step 3053 may be implemented through steps 531 to 533 , which are described in detail below.

[0206] In step 531, the high-frequency prediction module to be trained is used to predict the training rotation angles of the key bones and the training displacements of the root bones of the reference virtual object in each training animation frame of each training animation segment, and the predicted offset information of each clothing vertex of the virtual clothing in each training animation frame is obtained.

[0207] In some embodiments, based on the training displacement of the root bone of the virtual object in each training animation frame of each training animation segment, the training speed of the root bone of the virtual object in each training animation frame is determined, and then the training rotation angle of the key bone and the training speed of the root bone of the virtual object in each training animation frame are input into the high-frequency prediction module frame by frame to obtain the predicted offset information of each clothing vertex of the virtual clothing in each training animation frame.

[0208] In step 532 , the second predicted position of each garment vertex of the virtual garment in each training animation frame is determined using the second reference position of each garment vertex of the virtual garment in each training animation frame and the corresponding predicted offset information.

[0209] In some embodiments, the implementation process of step 532 is similar to the implementation process of step 334 , and step 532 can be implemented with reference to the implementation process of step 334 .

[0210] In step 533 , the high-frequency prediction module to be trained is trained using the second predicted positions of each garment vertex of the virtual garment in each training animation frame and the corresponding vertex label positions to obtain a trained high-frequency prediction module.

[0211] In some embodiments, the loss value used by the high-frequency prediction module during the training process is the sum of the third difference between the second predicted position of each garment vertex of the virtual garment and the vertex label position of each garment vertex on the virtual garment determined by offline physical cloth solution and the collision term. When determining the collision term, first determine the target point on the virtual garment that collides with the virtual object based on the second predicted position of each garment vertex of the virtual garment, that is, the point located inside the virtual object, and then determine the distance from the target point to the virtual object, and determine the sum of the distances of multiple target points to the virtual object as the collision term. The loss value of the high-frequency prediction module is back-propagated to the high-frequency prediction module, and the parameters of the high-frequency prediction model are adjusted using the gradient descent method until the training end condition is reached, thereby obtaining the trained high-frequency prediction module.

[0212] In the above steps 301 to 305, when training the clothing prediction model, the training data used is obtained through offline fabric solution, which can ensure the universality of the clothing prediction model after training obtained by using the training data, and has good prediction results for loose clothing and complex clothing. The quality of clothing motion prediction will be higher than the current real-time physical solution effect. In addition, when the complexity of the current pure physical solution model increases, the computing power requirements of the computer are high. In the embodiment of the present application, a deep learning model is used for clothing prediction, so the implementation scheme is more lightweight, and the clothing prediction model in the embodiment of the present application includes a high-frequency prediction module and a low-frequency prediction module. Compared with the related technology, by performing weighted summation on the action network under different parameters, the hardware consumption can be reduced, and the processing efficiency is high, which can meet the strong real-time requirements.

[0213] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0214] The embodiment of the present application provides a method for processing virtual object clothing data, which can be applied to a variety of live broadcast scenarios that require presenting virtual objects wearing virtual clothing, as well as offline video generation scenarios, gaming scenarios, virtual reality scenarios, augmented reality scenarios, and the like. In the embodiment of the present application, the method for processing virtual object clothing data is applied to a live broadcast scenario as an example. The anchor terminal receives configuration information for a virtual digital human live broadcast, wherein the configuration information includes the virtual digital human identifier for the live broadcast, the clothing identifier of the clothing worn by the virtual digital human during the live broadcast, and the like. After receiving the configuration information, the anchor terminal determines a clothing prediction model for virtual clothing prediction based on the clothing identifier. During the live broadcast, the host terminal receives live animation data collected by the motion capture device and transmits the live animation data to the game engine in the host terminal. The game engine calls the animation blueprint to obtain motion data from the live animation data. Based on the clothing prediction model, motion data, and attribute information of the virtual clothing, the game engine accurately predicts the motion state of the virtual digital human's virtual clothing when the virtual digital human executes the motion data, thereby improving the authenticity of the virtual digital human's live broadcast. The host terminal sends the live data stream of the virtual object wearing the virtual clothing to the server, and the server sends the live data to the viewer terminal, so that the live animation data is played through the viewer terminal. In addition, using the virtual object clothing data processing method provided in the embodiment of the present application, it is possible to quickly predict the animation of the virtual clothing during the live broadcast, with a quality close to that of offline solution, and it can avoid the solution errors that may be caused by physical solution during the live broadcast and the situation that cannot be recovered.

[0215] The method for processing the clothing data of virtual objects provided in the embodiments of the present application can also be applied to the rapid production of offline videos. After obtaining the animation data of the virtual object, the game engine in the terminal can configure the clothing identifier and virtual object identifier of the virtual clothing worn by the virtual object. After completing the configuration, the configuration information is used to determine the trained clothing prediction model, wherein the clothing prediction model is a neural network model (which can also be a deep learning model) for predicting the motion state of clothing. The trained clothing prediction model is used to predict and render the virtual clothing based on the animation data of the virtual object, so that when the virtual object executes the action corresponding to the animation data, the animation effect of the virtual clothing matching the action can be presented, thereby skipping the offline fabric solution process, quickly generating the video, and accelerating the video online.

[0216] Figure 13A is a schematic diagram of the effect of a certain frame rendered by offline solution of a single skirt; Figure 13B is an effect diagram of real-time prediction of the skirt using the trained clothing prediction model provided by the embodiment of the present application. By comparing Figure 13A and Figure 13B, it can be seen that the effect of rendering virtual clothing using the trained clothing prediction model provided by the embodiment of the present application is close to the effect of offline solution; Figure 13C and Figure 13D are two other effect diagrams predicted in real time in the UE engine using the trained clothing prediction model provided by the embodiment of the present application.

[0217] By using the virtual clothing rendering method provided in the embodiment of the present application, after obtaining the animation data of the virtual object, the fabric simulation is performed according to the animation data, and the result is rendered in real time on the screen. The virtual object clothing data processing method provided in the embodiment of the present application can be deployed in a game engine, for example, it can be deployed in a UE engine, a Unity engine, or other similar game engines. The virtual object clothing data processing method provided in the embodiment of the present application can improve the effect quality of real-time fabric simulation, or serve as an alternative to offline video fabric simulation, and can also reduce the performance overhead of the device and improve the robustness in the live broadcast scenario to avoid unacceptable errors.

[0218] FIG14 is a schematic diagram of another implementation flow of the method for processing clothing data of a virtual object provided in an embodiment of the present application, which will be described below in conjunction with FIG14 .

[0219] In step 401, the art-side terminal prepares data.

[0220] In some embodiments, data preparation by the art-side terminal includes: artists using the art terminal to model virtual object models, model clothing models, establish and bind skeletons in the virtual object models, and repair the acquired animation data. The art-side terminal obtains the virtual object models and clothing models in this business scenario, and obtains animation data. This animation data can be real-life character animation data recorded in a live broadcast scenario using a motion capture device, or it can be animation data obtained by redirecting existing animation data to the character model required for the business. The art-side terminal performs offline fabric solution based on the clothing model and animation data to obtain the offline fabric solution results.

[0221] For example, in a certain business scenario, 36,000 frames of data are collected for 20 minutes, including common actions in live broadcast scenes. After offline fabric solution, the offline fabric solution results obtained include at least the action data of the virtual object, the model data of the virtual object, the model data of the clothing model, the point cloud data of the virtual object in each animation frame, and the point cloud data of the clothing model in each animation frame.

[0222] The action data of a virtual object refers to the rotation angles of each key bone and the displacement of the root bone when the virtual object performs an action. The model data of a virtual object refers to the vertex information of the multiple triangular facets that make up the virtual object. The model data of a clothing model refers to the vertex information of the multiple triangular facets that make up the virtual clothing. The point cloud data of a virtual object refers to the position information and normal information of the vertices on each triangular facet when the virtual object performs an action. The point cloud data of a clothing model refers to the position information and normal information of the vertices on each triangular facet of the virtual clothing when the virtual object performs an action.

[0223] In step 402, data pre-processing is automated.

[0224] In some embodiments, the data preprocessing process includes extracting the rotation Euler angles of 52 key bones (key bones of the body other than fingers and toes) from the action data using 3D animation software (Maya) (the Euler angles will be converted into quaternions, rotation vectors, and rotation matrices for subsequent algorithm testing) and the displacement of the root bone; extracting the vertex positions and vertex normal values ​​of each vertex of the virtual object when the virtual object performs the action from the animation data of the virtual object; extracting the vertex positions of each vertex in the service model from the animation data of the clothing model, and using the displacement difference of each vertex between two frames as the velocity vector of each vertex. During the data preprocessing process, it is also necessary to determine the mean, standard deviation, normalization value, etc. of each of the above data and package the data.

[0225] Laplace smoothing is performed on the vertex position of each vertex of the clothing model and skin decomposition is performed to extract the position of each virtual bone of the clothing model and its skin weight.

[0226] In an embodiment of the present application, the data preprocessing in the BDMN algorithm is improved. Since BDMN uses a recurrent neural network, the information of the previous frame will be retained during each frame training. In actual model training, if each animation is directly sliced ​​into 200 frames and zero is padded when there are less than 200 frames, but in actual training, for example: for a 400-frame animation, if the action amplitude from the 200th frame to the 201st frame is large, and the skirt amplitude is large, the accumulated hidden layer weight value of the previous 200 frames will be lost at the 201st frame. When predicting the entire 400-frame animation, the accumulated hidden layer weight value of the previous 200 frames will be added at the 201st frame. The network will learn the inertia of the action, resulting in the skirt amplitude of the 201st frame, which should have been extremely large, being unable to swing to such a large amplitude due to the limitation of the hidden layer accumulation.

[0227] To solve this problem, the animation slicing method is changed in the embodiment of the present application. Each animation segment is still 200 frames long, but 100 overlapping image frames are added between each animation segment.

[0228] Figure 15 shows the clothing rendering effect at the 200-frame stage after a rotation animation is completed. 1501 on the left is the effect before improvement, and 1502 on the right is the effect after improvement. At this time, the hidden layer accumulation problem caused by data truncation in the model will be avoided to a certain extent.

[0229] In step 403, the model is trained.

[0230] In some embodiments, the deep learning model (corresponding to the clothing prediction model in other embodiments) includes a low-frequency prediction module and a high-frequency prediction module. When training the deep learning model, the low-frequency prediction module is trained first, and then the high-frequency prediction module is trained, and finally the trained deep learning model is obtained.

[0231] When training the low-frequency prediction module, the rotation data of the 22 key bones of the virtual object and the displacement value of the root bone are used as input data. According to the BDMN algorithm, the GRU is used as the recurrent unit to input the rotation data of the key bones of the virtual object and the displacement of the root bone frame by frame, thereby predicting the displacement and rotation angle of each virtual bone in the clothing model. Then, according to the location-based services (LBS) model, the predicted position of each vertex on the clothing model is determined according to the displacement and rotation angle of each virtual bone in the clothing model. The loss value is then determined using the predicted position of each vertex on the clothing model generated by LBS and the label position of each vertex on the clothing model determined by offline physical fabric solution and the preset loss function. The loss value is then back-propagated to the low-frequency prediction module to adjust the parameters of the low-frequency prediction module, and finally the trained low-frequency prediction module is obtained.

[0232] The loss function used by the low-frequency prediction module during training is the difference between the vertex position predicted by the low-frequency model and the vertex position calculated by actual physics plus the difference between the Laplace term of the vertex position predicted by the low-frequency prediction model and the Laplace term of the vertex calculated by actual physics.

[0233] In the high-frequency prediction module, instead of using a graph neural network to process the mesh of the clothing model obtained by the low-frequency prediction module, only a GRU is used to process the mesh of the clothing model obtained by the low-frequency prediction module. The input of the high-frequency prediction module is the rotation data of the 22 key bones of the virtual object and the displacement value of the root bone. The high-frequency prediction module predicts the offset value of each vertex in the clothing model. The predicted position of each vertex of the clothing model obtained by the low-frequency prediction module and the offset value of each vertex in the clothing model predicted by the high-frequency prediction module are spliced ​​together, and a multi-layer perceptron (MLP) is used to obtain the final predicted position of each vertex of the clothing model.

[0234] The loss function used in the high-frequency prediction module during training is the sum of the third difference between the final predicted position of each vertex on the garment model and the label position of each vertex on the garment model determined through offline physical fabric solving, and the collision term. To determine the collision term, the target point on the garment model that collides with the virtual object is first determined based on the final predicted position of each vertex on the garment model. The target point is also the point located inside the virtual object. The distance from the target point to the virtual object is then determined. The sum of the distances from multiple target points to the virtual object is used as the collision term.

[0235] In an embodiment of the present application, a deep learning network model consisting of a high-frequency prediction module and a low-frequency prediction module is used as a clothing prediction model, which can meet the requirements of low hardware consumption and strong real-time performance in actual projects.

[0236] Its single-machine prediction performance data is as follows:

[0237] Machine: 3080ti;

[0238] Model vertex count: 6788;

[0239] Number of virtual bones: 80;

[0240] Inference time: GPU idle, CPU idle: CPU: 8 milliseconds, with some jitter of 20-30 milliseconds; GPU: 6 milliseconds, with some jitter of 7-8 milliseconds.

[0241] In step 404, a plug-in is developed on the game engine side, which receives the trained deep learning model as input, uses the trained deep learning model to make real-time predictions on the animation data input from the game engine side, performs model penetration correction, connects to the game engine's material system, and renders the clothing in real time.

[0242] In some embodiments, the game engine can be a UE engine, or a Unity engine, etc. In the embodiments of the present application, the game engine is taken as an example of a UE engine. The architecture of the UE engine side is shown in Figure 16, including a data processing layer 1601, a prediction layer 1602, and a rendering layer 1603. Among them, the data processing layer 1601 is mainly used to implement processing processes such as data caching, model selection, action data processing, reading data, and hardware monitoring. The prediction layer 1602 is mainly used to implement processing processes such as data prediction, LBS model conversion, and collision detection. The rendering layer 1603 is mainly used to implement processing processes such as face generation, rendering cloth face patches, and calculating normals.

[0243] FIG17 is a schematic diagram of the implementation process of cloth simulation on the engine side provided by an embodiment of the present application. In some embodiments, step 404 can be implemented through the various steps shown in FIG17 , which are described in detail below.

[0244] In step 4041, object motion data is obtained.

[0245] In some embodiments, the object motion data may be obtained from live data collected in real time, or may be obtained from offline animation data.

[0246] When the object motion data is obtained from live broadcast data, the live broadcast data can be directly transmitted to the UE engine by the motion capture device, and the animation blueprint is called in the UE engine to obtain the rotation angle of the key bones of the virtual object in each animation frame and the world displacement of the root bone.

[0247] When the object motion data is obtained from offline animation data, the offline animation data can be imported into the UE engine, and the animation blueprint is also called in the UE engine to obtain the rotation angle of the key bones of the virtual object in each animation frame and the world displacement of the root bone.

[0248] In step 4042, prediction related data is read.

[0249] To ensure the accuracy of the deep learning model's predictions, it is necessary to read the model's configuration information. This information includes the type of deep learning model, the clothing model's type identification (ID), and the virtual object's type ID. Based on this configuration information, the corresponding deep learning model is retrieved. Furthermore, the skin weights and displacements of the clothing model's virtual skeleton, obtained during the data preprocessing phase, and the clothing model's facet information corresponding to the clothing model's type ID are also required.

[0250] In step 4043, hardware occupancy is monitored.

[0251] During the implementation of the virtual clothing rendering method provided in the embodiment of the present application, the hardware occupancy of the current device will be monitored, the current CPU and GPU occupancy rates will be read, and the task will be assigned to the CPU by default. If the CPU and GPU occupancy rates are both within an acceptable range, the task will be assigned according to the user's choice. If the GPU occupancy rate is low and the CPU occupancy rate is much greater than the GPU, the task will be assigned to the GPU.

[0252] In step 4044, the object motion data is processed.

[0253] The object motion data obtained in step 4041 cannot be directly input into the deep learning model. It needs to be converted into the right-hand coordinate system when training the deep learning model, and the rotation angle and root bone displacement of the key bones of the virtual object in each animation frame in the obtained motion data are normalized. Then, the speed of the root bone is determined based on the root bone displacement of the virtual object in each animation frame. The speed of the root bone is calculated as the root bone displacement of the previous frame minus the root bone displacement of the next frame.

[0254] In step 4045, virtual skeleton data is predicted.

[0255] In an embodiment of the present application, a local deployment method is adopted to integrate the deep learning model into the UE engine. The solution adopted is to read data frame by frame and export the trained deep learning model to the onnx format. The Unreal version of the OnnxRuntime plug-in is used to import the trained model into the UE. The exported deep learning model is a trained recurrent neural network. The trained deep learning model receives two inputs: one is the hidden layer data of the previous frame, and the other is the rotation angle of the key bones and the speed of the root bone of the current frame. The dimension of the hidden layer data here needs to be consistent with that during training.

[0256] In step 4046, the virtual skeleton is converted into model vertex data.

[0257] The low-frequency prediction module in the trained deep learning model predicts the rotation angle and position of the virtual bones in the clothing model. The location-based skeletal system (LBS) module then converts the rotation angle and position of the virtual bones into the coordinates of each vertex in the clothing model. In some embodiments, the decision to use the CPU's or GPU's LBS model to complete the conversion of virtual bones into model vertices is based on hardware monitoring results.

[0258] In step 4047, collision detection and correction are performed.

[0259] The collision detection correction here is mainly to correct the problem of penetration between the virtual object and the clothing model. In the embodiment of the present application, the collision volume of the virtual object is regarded as a combination of multiple capsules, and each vertex of the clothing model is subjected to collision detection with each capsule. First, the capsule is regarded as a cone, and the collision between the cone and the vertex is judged. If the current position of the vertex is inside the cone, the vertex is pushed out of the cone. The pushing direction is the direction of the line perpendicular to the edge of the cone through the vertex, and the pushing position is the intersection of the line perpendicular to the edge of the cone and the edge, that is, on the surface of the cone. Then judge the collision between the two spheres at the top of the capsule and the vertex. If the vertex is inside the sphere, it is pushed out of the sphere. The pushing direction is the direction connecting the vertex through the center of the sphere, and the pushing position is the intersection of the straight line connecting the vertex through the center of the sphere and the sphere, that is, on the surface of the sphere.

[0260] In step 4048, the engine renders the cloth.

[0261] Since the animation data is transmitted to the UE engine in real time, the deep learning model is required to be able to achieve real-time prediction and real-time rendering. At this time, it is necessary to dynamically generate meshes at runtime. In some embodiments, a procedural mesh component (PMC) is used to generate dynamic meshes. However, when using PMC, this mechanism cannot split normals, UVs or other attributes at the vertices. For a model, the number of vertices, the number of normals, and the number of UV coordinates on the vertices may not be equal. In short, a vertex may participate in the rendering of multiple meshes and may have multiple normals. The UV coordinates of vertices at the same position may also be different when rendering multiple meshes. In the embodiment of the present application, the normal data still records the vertex normals of each vertex, and for UV coordinates, if a vertex has two UV coordinates, it is regarded as two vertices and sent to the PMC, and the cloth rendering process is completed through the processing flow shown in Figure 18, which is explained in detail below.

[0262] As shown in Figure 18, the skeleton data of the virtual object is input into the deep learning model. Assume that 6788 vertex coordinate data are finally obtained, but the vertex data sent to the PMC is determined by the number of UV coordinates. When there are 6864 UV coordinates, the vertex coordinate data input to the PMC should also be 6864. At this time, a mapping table of 6788 vertices to 6864 UV data can be calculated in advance, and the vertices are mapped according to the index of the UV data. The 6788 vertices are mapped to 6864 vertices, and their vertex normal data are sent to the PMC for rendering.

[0263] In order to avoid the problem that there are obvious dividing lines between rendered patches and the overall model looks rough during rendering, in an embodiment of the present application, the vertex normals need to be recalculated. When a vertex belongs to different patches, the vertex may have different normal values. In an embodiment of the present application, the different normal values ​​of this vertex are reweighted and calculated as the vertex normal of this vertex. There are many weighting methods, such as weighting based on the angle, the size of the patch area, or hash clustering. Since the normal needs to be recalculated for each frame, the average weighted method with less performance consumption can be used, that is, all the normal values ​​of this vertex are calculated by arithmetic average to obtain the normal of this vertex.

[0264] In some embodiments, anti-aliasing processing may also be performed after step 4048. For high-speed moving meshes that change frame by frame, TAA is used in this embodiment to address aliasing issues, rather than the TSR anti-aliasing solution commonly used in UE engines. This prevents noise-like afterimages from remaining in the rendered cloth during high-speed motion.

[0265] In the rendering method of virtual clothing provided in the embodiment of the present application, real data is collected according to the business scenario of the live broadcast, offline data solution of the real data data set is performed, and a series of data processing is performed on it, and the simulation data required for the deep learning model is extracted for training to obtain the trained deep learning model. The trained deep learning model is integrated into the UE engine, the prediction process is accelerated in the UE engine, and the prediction results are rendered in real time, and the use of CPU or GPU for real-time rendering can be dynamically adjusted according to the load of the hardware device. After the prediction is completed, due to the lack of accuracy of the model prediction, a mold penetration problem may occur. To solve the mold penetration problem, the engine side will perform collision detection after obtaining the model prediction results, and push the mold penetration vertex outside the model through collision detection, solving the mold penetration problem to a certain extent. In this way, the whole set of plug-ins of the solution can be plug-in-and-play, which improves the portability of the solution, and can be solved in real time within the UE engine, can accept existing animation data and live broadcast data, and can render the effect in real time. Thanks to the learning of offline datasets, the quality of clothing motion prediction will be higher than that of current real-time physics-based solutions. Furthermore, the entire solution will be more lightweight, requiring less hardware resources than current real-time solutions. Due to the inherent advantages of data-driven approaches over physics-based solutions, the virtual clothing rendering method provided by the embodiments of this application will be more robust than current real-time solutions.

[0266] It is understandable that in the embodiments of the present application, when the action data of virtual objects, live broadcast data and other related data are applied to specific products or technologies, user permission or consent must be obtained, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0267] The following further describes an exemplary structure of a virtual object clothing data processing device 455 provided in an embodiment of the present application implemented as a software module. In some embodiments, as shown in FIG2 , the software modules stored in the virtual object clothing data processing device 455 in the memory 450 may include:

[0268] The first acquisition module 4551 is configured to obtain object action data and model configuration information of the virtual object; the second acquisition module 4552 is configured to obtain a trained clothing prediction model based on the model configuration information, wherein the trained clothing prediction model is trained using training action data of a reference virtual object and training point cloud data about virtual clothing, and the training action data and the training point cloud data are obtained by performing offline cloth solution on the training animation data; the prediction module 4553 is configured to use the clothing prediction model to perform prediction processing on the object action data to obtain vertex position information of each clothing vertex in the virtual clothing when the virtual object wears the virtual clothing and performs the object action data; the rendering module 4554 is configured to render the virtual clothing based on the vertex position information of each clothing vertex in the virtual clothing and the attribute information of the virtual clothing.

[0269] In some embodiments, the first acquisition module 4551 is further configured to: obtain animation data to be processed, where the animation data is live data collected in real time or pre-generated animation data; process the animation data using a game engine to obtain the rotation angle of the key bones preset in the virtual object and the displacement of the root bone of the virtual object in each animation frame of the animation data.

[0270] In some embodiments, the device also includes: a third acquisition module, configured to obtain the current CPU usage and GPU usage; a fourth acquisition module, configured to obtain processor selection information if the CPU usage is less than a first usage threshold and the GPU usage is less than a second usage threshold; a first determination module, configured to determine the processor corresponding to the processor selection information as the target processor.

[0271] In some embodiments, the device also includes: a second determination module, configured to determine the GPU as the target processor if the CPU usage is greater than or equal to a first usage threshold and the GPU usage is less than the second usage threshold; a third determination module, configured to determine the CPU as the target processor if the CPU usage is less than the first usage threshold and the GPU usage is greater than or equal to the second usage threshold; and a fourth determination module, configured to determine the processor with lower usage between the GPU and the CPU as the target processor if the CPU usage is greater than or equal to the first usage threshold and the GPU usage is greater than or equal to the second usage threshold.

[0272] In some embodiments, the prediction module 4553 is further configured to: normalize the rotation angles of the key bones preset in the virtual object in each animation frame and the displacement of the root bones of the virtual object, respectively, to obtain the normalized rotation angles of the key bones and the normalized displacements of the root bones of the virtual object in each animation frame; determine the speed of the root bones of the virtual object in each animation frame based on the normalized displacement of the root bones of the virtual object in each animation frame; use the trained clothing prediction model to predict the normalized rotation angles of the key bones and the speed of the root bones of the virtual object in each animation frame to obtain the vertex position information of each clothing vertex in the virtual clothing.

[0273] In some embodiments, the trained clothing prediction model includes a trained low-frequency prediction module and a trained high-frequency prediction module, and the prediction module 4553 is further configured to: use the low-frequency prediction module to predict the normalized rotation angle of the key bones and the speed of the root bones of the virtual object in each animation frame to obtain the first predicted rotation angle and first predicted position of the virtual bones of the virtual clothing in each animation frame; determine the first reference position of each clothing vertex of the virtual clothing in each animation frame according to the first predicted rotation angle, first predicted position and skin weight of the virtual bones of the virtual clothing in each animation frame; use the high-frequency prediction module to predict the normalized rotation angle of the key bones and the speed of the root bones of the virtual object in each animation frame to obtain the vertex offset information of each clothing vertex of the virtual clothing in each animation frame; superimpose the first reference position of each clothing vertex of the virtual clothing in each animation frame with the corresponding vertex offset information to obtain the vertex position information of each clothing vertex of the virtual clothing in each animation frame.

[0274] In some embodiments, the rendering module 4554 is further configured to: obtain a capsule model corresponding to the virtual object;

[0275] Based on the vertex position information of each clothing vertex in the virtual clothing and the capsule model, collision detection and correction are performed on each clothing vertex in the virtual clothing to obtain the corrected vertex position information of each clothing vertex; vertex normals of each clothing vertex are obtained; and based on the corrected vertex position information of each clothing vertex, the vertex normals and the texture coordinates, the virtual clothing is rendered.

[0276] In some embodiments, the rendering module 4554 is further configured to: determine the center line of the capsule body, determine the vertical distance between each clothing vertex and the center line based on the vertex position information of each clothing vertex; determine the clothing vertex whose vertical distance is less than the reference distance as the vertex to be corrected that collides; determine the difference between the reference distance and the vertical distance as the offset distance of the vertex to be corrected; move the vertex position information of the vertex to be corrected by the offset distance in the opposite direction of the perpendicular line from the vertex to be corrected to the center line to obtain the corrected vertex position information of the vertex to be corrected.

[0277] In some embodiments, the rendering module 4554 is further configured to: determine the triangular facets where each clothing vertex is located; when a clothing vertex exists in only one triangular facet, determine the vertex normal of the clothing vertex based on the corrected vertex position information of the three clothing vertices of the triangular facet; when a clothing vertex exists in at least two triangular facets, determine the vertex normal of the clothing vertex in each triangular facet; perform weighted calculation on the vertex normal of the clothing vertex in each triangular facet to obtain the vertex normal of the clothing vertex.

[0278] In some embodiments, the rendering module 4554 is further configured to: determine the area of ​​each triangular facet, and determine the weight corresponding to each triangular facet based on the area of ​​each triangular facet; and perform weighted processing on the vertex normals of the clothing vertices in each triangular facet based on the weight corresponding to each triangular facet to obtain the vertex normals of the clothing vertices.

[0279] In some embodiments, the device also includes: a fifth acquisition module, configured to acquire training animation data and a clothing prediction model to be trained; a cloth solution module, configured to perform offline cloth solution on the training animation data to obtain training action data and training point cloud data of the virtual clothing; a first extraction module, configured to extract the training rotation angles of the key bones and the training displacements of the root bones of the reference virtual object in each training animation frame from the training action data; a second extraction module, configured to extract the vertex label positions of the clothing vertices of the virtual clothing in each training animation frame from the training point cloud data of the virtual clothing; a first training module, configured to use the training rotation angles of the key bones and the training displacements of the root bones of the reference virtual object in each training animation frame, as well as the vertex label positions of the clothing vertices of the virtual clothing in each training animation frame, to train the clothing prediction model to be trained to obtain a trained clothing prediction model.

[0280] In some embodiments, the clothing prediction model to be trained includes at least a low-frequency prediction module to be trained and a high-frequency prediction module to be trained. The first training module is further configured to: use the training rotation angles of the key bones and the training displacements of the root bones of the reference virtual object in each training animation frame, and the vertex label positions of the clothing vertices of the virtual clothing in each training animation frame to train the low-frequency prediction module to obtain a trained low-frequency prediction module; use the trained low-frequency prediction module, the training rotation angles of the key bones and the training displacements of the root bones of the virtual object in each training animation frame of each training animation segment to determine the second reference position of each clothing vertex of the virtual clothing in each training animation frame; use the training rotation angles of the key bones and the training displacements of the root bones of the reference virtual object in each training animation frame of each training animation segment, the vertex label positions of the clothing vertices of the virtual clothing in each training animation frame, and the second reference positions to train the high-frequency prediction module to obtain a trained high-frequency prediction module.

[0281] In some embodiments, the first training module is further configured to: determine the bone label rotation angle, bone label position and skin weight of the virtual skeleton of the virtual garment in each training animation frame based on the vertex label position of the garment vertices of the virtual garment in each training animation frame; divide the multiple training animation frames in the training animation data into multiple training animation segments, and two adjacent training animation segments include a preset number of overlapping training animation frames; use the low-frequency prediction module to be trained to predict the training rotation angle of the key bones and the training displacement of the root bones of the reference virtual object in each training animation frame of each training animation segment to obtain the second predicted rotation angle and second predicted position of the virtual skeleton of the virtual garment in each training animation frame; determine the third reference position of each garment vertex of the virtual garment in each training animation frame according to the second predicted rotation angle, second predicted position and skin weight of the virtual skeleton of the virtual garment in each training animation frame; use the third reference position and vertex label position of each garment vertex of the virtual garment in each training animation frame to train the low-frequency prediction module to obtain a trained low-frequency prediction module.

[0282] In some embodiments, the first training module is further configured to: use the high-frequency prediction module to be trained to predict the training rotation angles of the key bones and the training displacements of the root bones of the reference virtual object in each training animation frame of each training animation segment, and obtain the predicted offset information of each clothing vertex of the virtual clothing in each training animation frame; use the second reference position of each clothing vertex of the virtual clothing in each training animation frame and the corresponding predicted offset information to determine the second predicted position of each clothing vertex of the virtual clothing in each training animation frame; use the second predicted position of each clothing vertex of the virtual clothing in each training animation frame and the corresponding vertex label position to train the high-frequency prediction module to obtain a trained high-frequency prediction module.

[0283] An embodiment of the present application provides a computer program product, comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the method for processing virtual object clothing data described in the embodiment of the present application.

[0284] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the virtual object clothing data processing method provided in the embodiment of the present application, for example, the virtual object clothing data processing method shown in Figures 3 and 14.

[0285] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0286] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0287] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, e.g., in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0288] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0289] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A method for processing clothing data of a virtual object, applied to an electronic device, the method comprising: Obtain object action data and model configuration information of the virtual object; Acquire a trained clothing prediction model based on the model configuration information, wherein the trained clothing prediction model is obtained by training with reference to training action data of a virtual object and training point cloud data about virtual clothing, wherein the training action data and the training point cloud data are obtained by performing offline cloth solving on training animation data; Using the clothing prediction model to perform prediction processing on the object action data, to obtain vertex position information of each clothing vertex in the virtual clothing when the virtual object wears the virtual clothing and performs the object action data; The virtual clothing is rendered based on vertex position information of each clothing vertex in the virtual clothing and attribute information of the virtual clothing.

2. The method according to claim 1, wherein: The step of obtaining the object action data of the virtual object includes: Acquire animation data to be processed, wherein the animation data is live animation data collected in real time or pre-generated animation data; The animation data is processed by using a game engine to obtain the rotation angle of the preset key bones of the virtual object and the displacement of the root bone of the virtual object in each animation frame of the animation data.

3. The method according to claim 1, wherein: Before performing prediction processing on the object action data using the clothing prediction model to obtain vertex position information of each clothing vertex in the virtual clothing when the virtual object wears the virtual clothing to perform the object action data, the method further includes: Get the current CPU usage and GPU usage; If the CPU usage is less than a first usage threshold, and the GPU usage is less than a second usage threshold, obtaining processor selection information; The processor corresponding to the processor selection information is determined as the target processor, wherein the target processor is used to perform prediction processing on the action data using the trained clothing prediction model to obtain vertex position information of each clothing vertex in the virtual clothing when the virtual object wears the virtual clothing and executes the object action data.

4. The method according to claim 3, wherein: The method further comprises: If the CPU usage is greater than or equal to a first usage threshold, and the GPU usage is less than the second usage threshold, determining the GPU as a target processor; If the CPU usage is less than a first usage threshold, and the GPU usage is greater than or equal to the second usage threshold, the CPU is determined as a target processor; If the CPU usage is greater than or equal to the first usage threshold, and the GPU usage is greater than or equal to the second usage threshold, a processor with a lower usage among the GPU and the CPU is determined as the target processor.

5. The method according to claim 2, wherein: The step of using the clothing prediction model to predict the action data to obtain vertex position information of each clothing vertex in the virtual clothing when the virtual object wears the virtual clothing and performs the object action data includes: Normalizing the rotation angle of the preset key bones of the virtual object and the displacement of the root bones of the virtual object in each animation frame to obtain the normalized rotation angle of the key bones and the normalized displacement of the root bones of the virtual object in each animation frame; Determining the speed of the root bone of the virtual object in each animation frame based on the normalized displacement of the root bone of the virtual object in each animation frame; The clothing prediction model is used to predict the normalized rotation angle of the key bones and the speed of the root bones of the virtual object in each animation frame, and the virtual object wearing the virtual clothing performs the object action. The data is the vertex position information of each garment vertex in the virtual garment.

6. The method according to claim 5, wherein: The clothing prediction model includes a low-frequency prediction module and a high-frequency prediction module. The clothing prediction model is used to predict the normalized rotation angles of the key bones and the speeds of the root bones of the virtual object in each animation frame to obtain the vertex position information of each clothing vertex in the virtual clothing when the virtual object wears the virtual clothing and performs the object action data, including: The low-frequency prediction module is used to predict the normalized rotation angle of the key bones and the speed of the root bones of the virtual object in each animation frame to obtain a first predicted rotation angle and a first predicted position of the virtual bones of the virtual clothing in each animation frame; Determining a first reference position of each garment vertex of the virtual garment in each animation frame according to a first predicted rotation angle, a first predicted position and a skin weight of a virtual skeleton of the virtual garment in each animation frame; The high-frequency prediction module is used to predict the normalized rotation angle of the key bones and the speed of the root bones of the virtual object in each animation frame, and obtain the vertex offset information of each clothing vertex of the virtual clothing in each animation frame; The first reference position of each garment vertex of the virtual garment in each animation frame is superimposed with the corresponding vertex offset information to obtain the vertex position information of each garment vertex of the virtual garment in each animation frame.

7. The method according to any one of claims 1 to 6, wherein: The attribute information of the virtual clothing at least includes the texture coordinates of each clothing vertex, and the rendering of the virtual clothing based on the vertex position information of each clothing vertex in the virtual clothing and the attribute information of the virtual clothing includes: Obtaining a capsule model corresponding to the virtual object; Based on the vertex position information of each clothing vertex in the virtual clothing and the capsule model, performing collision detection correction on each clothing vertex in the virtual clothing to obtain the corrected vertex position information of each clothing vertex; Obtaining vertex normals of each of the garment vertices; The virtual clothing is rendered based on the corrected vertex position information of each clothing vertex, the vertex normal and the texture coordinates.

8. The method according to claim 7, wherein: The method of performing collision detection and correction on each clothing vertex in the virtual clothing based on the vertex position information of each clothing vertex in the virtual clothing and the capsule model to obtain the corrected vertex position information of each clothing vertex includes: Determine a center line of the capsule model, and determine a vertical distance between each of the garment vertices and the center line based on vertex position information of each of the garment vertices; The clothing vertices whose vertical distances are smaller than the reference distance are determined as vertices to be corrected that collide; Determine the difference between the reference distance and the vertical distance as the offset distance of the vertex to be corrected; The vertex position information of the vertex to be corrected is moved by the offset distance in the opposite direction of the perpendicular line from the vertex to be corrected to the midline to obtain the corrected vertex position information of the vertex to be corrected.

9. The method according to claim 7, wherein: The step of obtaining the vertex normal of each of the garment vertices comprises: For each garment vertex, determining the triangular facet where the garment vertex is located; When the garment vertex exists only in one triangular facet, determining the vertex normal of the garment vertex based on the corrected vertex position information of three garment vertices of the triangular facet; When a clothing vertex exists in at least two triangular facets, determining a vertex normal of the clothing vertex in each triangular facet; The vertex normals of the clothing vertices on each triangular facet are weighted to obtain the vertex normals of the clothing vertices.

10. The method according to claim 9, wherein: The step of performing weighted processing on the vertex normals of the clothing vertices on each triangular facet to obtain the vertex normals of the clothing vertices includes: Determine the area of ​​each triangular facet, and determine the weight corresponding to each triangular facet based on the area of ​​each triangular facet; Based on the weight corresponding to each triangular facet, weighted processing is performed on the vertex normals of the clothing vertices on each triangular facet to obtain the vertex normals of the clothing vertices.

11. The method according to any one of claims 1 to 6, wherein: The method further comprises: Obtain training animation data and clothing prediction model to be trained; Performing offline cloth solving on the training animation data to obtain training motion data and training point cloud data of virtual clothing; Extracting the training rotation angles of the key bones and the training displacements of the root bones of the reference virtual object in each training animation frame from the training motion data; Extracting vertex label positions of clothing vertices of the virtual clothing in each training animation frame from the training point cloud data of the virtual clothing; The clothing prediction model is trained using the training rotation angles of the key bones and the training displacements of the root bones of the reference virtual object in each training animation frame, as well as the vertex label positions of the clothing vertices of the virtual clothing in each training animation frame, to obtain a trained clothing prediction model.

12. The method according to claim 11, wherein: The clothing prediction model to be trained includes at least a low-frequency prediction module to be trained and a high-frequency prediction module to be trained. The method of training the clothing prediction model by using the training rotation angles of the key bones and the training displacements of the root bones of the reference virtual object in each training animation frame, and the vertex label positions of the clothing vertices of the virtual clothing in each training animation frame, to obtain the trained clothing prediction model, comprises: Using the training rotation angles of the key bones and the training displacements of the root bones of the reference virtual object in each training animation frame, and the vertex label positions of the clothing vertices of the virtual clothing in each training animation frame, the low-frequency prediction module to be trained is trained to obtain a trained low-frequency prediction module; Determine the second reference position of each clothing vertex of the virtual clothing in each training animation frame by using the trained low-frequency prediction module, the training rotation angle of the key bone and the training displacement of the root bone of the reference virtual object in each training animation frame of each training animation segment, wherein the training animation segment is obtained by dividing a plurality of training animation frames, and two adjacent training animation segments include a preset number of overlapping training animation frames; The high-frequency prediction module to be trained is trained by using the training rotation angles of the key bones and the training displacements of the root bones of the reference virtual object in each training animation frame of each training animation segment, the vertex label positions of the clothing vertices of the virtual clothing in each training animation frame and the second reference position to obtain a trained high-frequency prediction module.

13. The method according to claim 12, wherein: The low-frequency prediction module to be trained is trained by using the training rotation angles of the key bones and the training displacements of the root bones of the reference virtual object in each training animation frame, and the vertex label positions of the clothing vertices of the virtual clothing in each training animation frame, to obtain the trained low-frequency prediction module, including: Determine the bone label rotation angle, bone label position and skin weight of the virtual skeleton of the virtual clothing in each training animation frame based on the vertex label positions of the clothing vertices of the virtual clothing in each training animation frame; Dividing a plurality of training animation frames in the training animation data into a plurality of training animation segments; Using the low-frequency prediction module to be trained, the training rotation angle of the key bones and the training displacement of the root bones of the reference virtual object in each training animation frame of each training animation segment are predicted to obtain a second predicted rotation angle and a second predicted position of the virtual bones of the virtual clothing in each training animation frame; Determine the third reference position of each clothing vertex of the virtual clothing in each training animation frame according to the second predicted rotation angle, the second predicted position and the skin weight of the virtual skeleton of the virtual clothing in each training animation frame. Place; The low-frequency prediction module to be trained is trained by using the third reference position and the vertex label position of each clothing vertex of the virtual clothing in each training animation frame to obtain a trained low-frequency prediction module.

14. The method according to claim 12, wherein: The method uses the training rotation angles of the key bones and the training displacements of the root bones of the reference virtual object in each training animation frame of each training animation segment, the vertex label positions of the clothing vertices of the virtual clothing in each training animation frame and the second reference position to train the high-frequency prediction module to be trained, and obtains the trained high-frequency prediction module, including: Using the high-frequency prediction module to be trained, the training rotation angle of the key bones and the training displacement of the root bones of the reference virtual object in each training animation frame of each training animation segment are predicted, so as to obtain the prediction offset information of each clothing vertex of the virtual clothing in each training animation frame; Determine the second predicted position of each garment vertex of the virtual garment in each training animation frame by using the second reference position of each garment vertex of the virtual garment in each training animation frame and the corresponding predicted offset information; The high-frequency prediction module to be trained is trained using the second predicted position of each clothing vertex of the virtual clothing in each training animation frame and the corresponding vertex label position to obtain a trained high-frequency prediction module.

15. A device for processing clothing data of a virtual object, the device comprising: A first acquisition module is configured to acquire object action data and model configuration information of a virtual object; a second acquisition module, configured to acquire a trained clothing prediction model based on the model configuration information, wherein the trained clothing prediction model is obtained by training with reference to training action data of a virtual object and training point cloud data about virtual clothing, wherein the training action data and the training point cloud data are obtained by performing offline cloth solving on the training animation data; A prediction module, configured to use the clothing prediction model to perform prediction processing on the object action data to obtain vertex position information of each clothing vertex in the virtual clothing when the virtual object wears the virtual clothing and performs the object action data; The rendering module is configured to render the virtual clothing based on vertex position information of each clothing vertex in the virtual clothing and attribute information of the virtual clothing.

16. An electronic device, comprising: A memory for storing computer executable instructions; A processor, configured to implement the method according to any one of claims 1 to 14 when executing the computer executable instructions stored in the memory.

17. A computer-readable storage medium storing computer-executable instructions or a computer program, wherein the computer-executable instructions or the computer program, when executed by a processor, implement the method according to any one of claims 1 to 14.

18. A computer program product, comprising computer executable instructions or a computer program, wherein the computer executable instructions or the computer program, when executed by a processor, implements the method according to any one of claims 1 to 14.

Images