Training of neural network model and role control method and device, electronic equipment

Abstract

The application provides a neural network model training and role control method and device, and electronic equipment; the method comprises the following steps: obtaining a first state vector corresponding to a virtual role in a virtual scene at a first time, and obtaining a second state vector corresponding to the virtual role at a second time; based on the first state vector and the second state vector, calling a first neural network model to perform action prediction processing to obtain a first action instruction, and calling an action control model to perform action prediction processing to obtain a second action instruction; selecting a target action instruction for controlling the virtual role from the first action instruction and the second action instruction based on a selection probability; combining the first state vector, the second state vector, and the second action instruction into a sample, and storing the sample into a sample queue; and training the first neural network model based on multiple samples in the sample queue. Through the application, accurate control of the action of the virtual role can be realized in a resource-intensive manner.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, and electronic device for training and controlling a neural network model. Background Technology

[0002] Virtual characters in games typically exhibit diverse and agile movements, such as running and jumping. In related technologies, these movements are captured by professional stunt performers to form an animation asset library. During gameplay, the closest animation is matched in real-time from this library based on the character's state and player commands. However, the memory and computational demands of this method increase linearly with the amount of animation assets.

[0003] In addition, related technologies also provide rule-based dynamic control methods, such as whole-body control (WBC) and model predictive control (MPC). However, the disadvantage of rule-based methods is that computational efficiency and high dynamic accuracy cannot be achieved simultaneously. In order to improve the accuracy, a lot of computing resources are required, which may cause animation stuttering when deployed online in games. Summary of the Invention

[0004] This application provides a method for training a neural network model, a method for controlling a character, a device, an electronic device, a computer-readable storage medium, and a computer program product, which can achieve precise control of the actions of a virtual character in a resource-efficient manner.

[0005] The technical solution of this application embodiment is implemented as follows: This application provides a method for training a neural network model, including: Obtain the first state vector of the virtual character in the virtual scene at the first moment, and obtain the second state vector of the virtual character at the second moment; Based on the first state vector and the second state vector, a first neural network model is invoked to perform action prediction processing to obtain a first action instruction, and an action control model is invoked to perform action prediction processing to obtain a second action instruction. The computational complexity of the first neural network model is less than that of the action control model. Based on the selection probability, a target action instruction for controlling the virtual character is selected from the first action instruction and the second action instruction, wherein the selection probability gradually increases during the training process of the first neural network model; The first state vector, the second state vector, and the second action instruction are combined into a sample, and the sample is stored in a sample queue. The first neural network model is trained based on multiple samples in the sample queue, wherein the first neural network model is used to replace the motion control model to control the actions of the virtual character.

[0006] This application provides a training apparatus for a neural network model, comprising: The acquisition module is used to acquire the first state vector corresponding to the virtual character in the virtual scene at the first moment, and to acquire the second state vector that the virtual character needs to correspond to at the second moment. The action prediction module is used to call a first neural network model to perform action prediction processing to obtain a first action instruction based on the first state vector and the second state vector, and to call an action control model to perform action prediction processing to obtain a second action instruction, wherein the computational complexity of the first neural network model is less than that of the action control model. A selection module is used to select a target action instruction for controlling the virtual character from the first action instruction and the second action instruction based on a selection probability, wherein the selection probability gradually increases during the training process of the first neural network model; The combination module is used to combine the first state vector, the second state vector, and the second action instruction into a sample; A storage module is used to store the samples into a sample queue; The training module is used to train the first neural network model based on multiple samples in the sample queue, wherein the first neural network model is used to replace the motion control model to control the actions of the virtual character.

[0007] This application provides a method for role control of a neural network model, including: Obtain the fourth state vector of the virtual character in the virtual scene at the fourth moment, and the control command triggered for the virtual character; Based on the control instruction vector corresponding to the control instruction and the fourth state vector, determine the fifth state vector that the virtual character needs to correspond to at the fifth moment; Based on the fourth state vector and the fifth state vector, the first neural network model is invoked to perform action prediction processing to obtain the third action instruction, and the action of the virtual character is controlled according to the third action instruction; The first neural network model is trained according to the neural network model training method provided in the embodiments of this application.

[0008] This application provides a role control device for a neural network model, including: The acquisition module is used to acquire the fourth state vector of the virtual character in the virtual scene at the fourth moment, as well as the control command triggered on the virtual character; The determining module is used to determine the fifth state vector that the virtual character needs to correspond to at the fifth moment, based on the control instruction vector corresponding to the control instruction and the fourth state vector. The action prediction module is used to call the first neural network model to perform action prediction processing based on the fourth state vector and the fifth state vector to obtain the third action command; The first neural network model is trained according to the neural network model training method provided in the embodiments of this application; The control module is used to control the actions of the virtual character according to the third action command.

[0009] This application provides an electronic device, including: Memory, used to store executable instructions; The processor, when executing executable instructions stored in the memory, implements the training method or role control method of the neural network model provided in the embodiments of this application.

[0010] This application provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the training method or role control method of the neural network model provided in this application.

[0011] This application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the training method or role control method of the neural network model provided in this application.

[0012] The embodiments of this application have the following beneficial effects: By continuously increasing the probability of selecting the first action command from the first action command of the first neural network model and the second action command of the action control model during the training process of the first neural network model, the first neural network model with low computational complexity can continuously fit the control capability of the action control model with high computational complexity. Thus, when the first neural network model replaces the action control model to control the action of the virtual character, it can achieve at least the same control accuracy as the action control model with the lowest possible consumption of computational resources and time. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the architecture of the neural network model training and role control system 100 provided in the embodiments of this application; Figure 2A This is a schematic diagram of the structure of the server 200 provided in the embodiments of this application; Figure 2B This is a schematic diagram of the structure of the terminal device 400 provided in the embodiments of this application; Figure 3 This is a flowchart illustrating the training method for the neural network model provided in the embodiments of this application; Figure 4 This is a flowchart illustrating the training method for the neural network model provided in the embodiments of this application; Figure 5 This is a flowchart illustrating the training method for the neural network model provided in the embodiments of this application; Figure 6 This is a flowchart illustrating the role control method for a neural network model provided in an embodiment of this application; Figure 7 This is a schematic diagram of the training process of the neural network model provided in the embodiments of this application; Figure 8 This is a schematic diagram of the testing process of the neural network model provided in the embodiments of this application; Figure 9 This is a flowchart illustrating the training method for the neural network model provided in the embodiments of this application; Figure 10 This is a schematic diagram of a bipedal character model in Unreal Engine 4 provided in this application embodiment. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0016] It is understood that in the embodiments of this application, data related to user information (such as user-inputted control commands) is involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0017] In the following description, the terms “first, second, ...” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first, second, ...” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0019] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0020] 1) Neural Networks (NN): A complex network system formed by the extensive interconnection of a large number of simple processing units (called neurons). It reflects many basic characteristics of human brain function and is a highly complex nonlinear dynamic learning system. Neural network models have the capabilities of large-scale parallelism, distributed storage and processing, self-organization, self-adaptation, and self-learning. They are particularly suitable for handling imprecise and fuzzy information processing problems that require simultaneous consideration of many factors and conditions.

[0021] 2) Virtual Scene: This refers to the scene displayed (or provided) by the application when it runs on the terminal device. A virtual scene can be a simulation of the real world, a semi-simulated / semi-fictional virtual environment, or a purely fictional virtual environment. A virtual scene can be any of a two-dimensional, 2.5-dimensional, or three-dimensional virtual scene; this application does not limit the dimension of the virtual scene. For example, a virtual scene may include the sky, land, ocean, etc., and the land may include environmental elements such as deserts and cities. The user can control a virtual character to move within this virtual scene.

[0022] 3) Virtual Characters: These are the images of various people and objects that can interact with a virtual scene, or movable objects within the virtual scene. These movable objects can be virtual characters, virtual animals, anime characters, etc., such as people or animals displayed in a virtual scene. A virtual character can be a virtual avatar representing the user within the virtual scene. A virtual scene can include multiple virtual characters, each with its own shape and volume, occupying a portion of the space within the virtual scene.

[0023] 4) Queue: A special type of linear list that only allows deletion operations at the front (front) and insertion operations at the rear (rear). Like a stack, a queue is a linear list with restricted operations. Because a queue only allows insertion at one end and deletion at the other, only the earliest sample that entered the queue can be deleted from the queue first. Therefore, a queue is also called a first-in, first-out (FIFO) linear list.

[0024] 5) Unreal Engine: A game engine refers to the core components of pre-written, editable computer game systems or interactive real-time graphics applications. These systems provide game designers with various tools needed to write games, aiming to allow them to easily and quickly create game programs without starting from scratch. Simply put, a game engine is a set of code frameworks written by developers, and game developers use these code frameworks to quickly implement game development.

[0025] 6) Rule-based Model: A model that calculates motion commands based on motion control rules (e.g., positive dynamics equations), such as WBC controllers and MPC controllers. These two types of controllers perform physical dynamics modeling of the virtual character, using mathematical optimization to calculate the character's expected position and posture at the next moment, based on the character's current state and user input, as well as the control commands required to each joint to achieve this state. The character's state can include its position, posture, velocity information, and the rotation angle and rotational speed of each joint. Joint commands include the target rotation angle, target rotational speed, feedforward torque, spring parameters, and damping parameters required by the proportional-derivative (PD) controller.

[0026] The applicant discovered that rule-based skeletal character dynamics control methods (such as WBC controllers and MPC controllers) in related technologies have the drawback that computational efficiency and high dynamic accuracy are mutually exclusive. This is determined by the derivation of mathematical optimization theory. Rule-based methods require a large amount of computational resources, which may cause animation stuttering when deployed online in games. If computational efficiency is prioritized, for WBC controllers, approximations are needed in the optimization calculations, such as omitting higher-order terms in Taylor expansions or reducing the number of iterations when solving differential equations using iterative methods; for MPC controllers, the step size of the prediction calculations needs to be shortened. Both of these measures undoubtedly reduce the computational accuracy of the control strategy. For character movements with low dynamic characteristics, such processing has little impact; however, for high-dynamic movements, such as running, martial arts, and professional sports movements, this will cause severe deformation and distortion of the character's movements, resulting in loss of control.

[0027] Furthermore, the applicant discovered that after calculating the character's expected position and posture at the next moment based on the character's current state and user input, directly assigning these values ​​results in kinematic skeletal animation generation. However, animations generated in this way do not necessarily conform to the laws of dynamics, thus potentially leading to animation distortion and an inability to handle object collisions and terrain undulations in the game environment.

[0028] In view of this, embodiments of this application provide a method, apparatus, electronic device, computer-readable storage medium, and computer program product for training neural network models and controlling virtual characters, which can achieve precise control of virtual character actions in a resource-efficient manner. The exemplary application of the electronic device provided in the embodiments of this application is described below. The electronic device provided in the embodiments of this application can be implemented as a server, or implemented collaboratively by a server and terminal devices.

[0029] The following description uses the role control method of the neural network model provided in the embodiments of this application, which is implemented collaboratively by a server and a terminal device, as an example.

[0030] For example, see Figure 1 , Figure 1 This is a schematic diagram of the architecture of the neural network model training and character control system 100 provided in this application embodiment. It aims to support an application capable of achieving precise control of virtual character movements in a resource-efficient manner, such as... Figure 1 As shown, the neural network model training and role control system 100 includes a server 200, a network 300, and a terminal device 400. A client 410 runs on the terminal device 400. The client 410 can be various types of applications supporting virtual scenes, such as any of the following: first-person shooter (FPS) games, third-person shooter games, virtual reality applications, 3D map programs, or multiplayer shooting survival games. It can also be a game engine client (e.g., a UE4 client). The terminal device 400 connects to the server 200 via the network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both.

[0031] In some embodiments, server 200 can train a first neural network model based on multiple samples stored in a sample queue, and send the trained first neural network model to terminal device 400 via network 300, so that terminal device 400 uses the trained first neural network model instead of motion control model to control the virtual character's movements. In this way, by using a first neural network model with low computational complexity to fit a motion control model with high computational complexity, precise control of the virtual character's movements can be achieved with the lowest possible computational resources and time, resulting in smooth and realistic high-quality animation and improving the user's visual experience.

[0032] In other embodiments, the embodiments of this application can also be implemented with the aid of cloud technology, which refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to realize the computation, storage, processing, and sharing of data.

[0033] Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form resource pools, allowing for on-demand use with flexibility and convenience. Cloud computing technology will become a crucial support. The backend services of cloud computing systems require substantial computing and storage resources.

[0034] Example, Figure 1 The server 200 shown can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal device 400 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, in-vehicle terminal, etc., but is not limited to these. The terminal device 400 and the server 200 can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0035] The following continues... Figure 1 The structure of server 200 shown in the diagram will be explained. See also... Figure 2A , Figure 2A This is a schematic diagram of the structure of the server 200 provided in the embodiments of this application. Figure 2AThe server 200 shown includes at least one processor 210, memory 240, and at least one network interface 220. The various components of server 200 are coupled together via a bus system 230. It is understood that the bus system 230 is used to implement communication between these components. In addition to a data bus, the bus system 230 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2A The general labeled all buses as Bus System 230.

[0036] Processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0037] The memory 240 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 240 may optionally include one or more storage devices physically located away from the processor 210.

[0038] The memory 240 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 240 described in this application embodiment is intended to include any suitable type of memory.

[0039] In some embodiments, memory 240 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0040] Operating system 241 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; The network communication module 242 is used to reach other computing devices via one or more (wired or wireless) network interfaces 220, such as Bluetooth, WiFi, and Universal Serial Bus (USB). In some embodiments, the training apparatus for the neural network model provided in this application can be implemented in software. Figure 2A A training device 243 for a neural network model stored in memory 240 is shown. This device can be software in the form of programs and plug-ins, including the following software modules: acquisition module 2431, action prediction module 2432, selection module 2433, combination module 2434, storage module 2435, training module 2436, determination module 2437, splicing module 2438, construction module 2439, transformation module 24310, and input module 24311. These modules are logically connected and can therefore be arbitrarily combined or further split according to their implemented functions. It should be noted that... Figure 2A For ease of explanation, all the above modules are shown at once, but this should not be considered as excluding the implementation of the neural network model training device 243, which may only include the acquisition module 2431, action prediction module 2432, selection module 2433, combination module 2434, storage module 2435 and training module 2436. The functions of each module will be described below.

[0041] The following continues... Figure 1 The structure of the terminal device 400 shown in the diagram will be described. See also... Figure 2B , Figure 2B This is a schematic diagram of the structure of the terminal device 400 provided in the embodiments of this application, as shown below. Figure 2B As shown, the terminal device 400 includes: at least one network interface 420, a user interface 430, a system bus 440, a memory 450, and a processor 460. The user interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touchscreen display, camera, other input buttons, and controls. The memory 450 includes: an operating system 451, a network communication module 452, a presentation module 453 for enabling the display of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 associated with the user interface 430 (e.g., displays, speakers, etc.), an input processing module 454 for detecting and translating one or more user inputs or interactions from one or more input devices 432, and a neural network model-based role control device 455. The role control device 455 of the neural network model includes: an acquisition module 4551, a determination module 4552, an action prediction module 4553, and a control module 4554. These modules are logically related and can be arbitrarily combined or further divided according to the functions they implement. The functions of each module will be described below.

[0042] The training method of the neural network model provided in this application embodiment will be described below with reference to the exemplary application and implementation of the server provided in the embodiments of this application.

[0043] See Figure 3 , Figure 3 This is a flowchart illustrating the training method for the neural network model provided in this application embodiment, which will be combined with... Figure 3 The steps shown are explained.

[0044] In step 101, the first state vector corresponding to the virtual character in the virtual scene at the first moment is obtained, and the second state vector corresponding to the virtual character at the second moment is obtained.

[0045] In some embodiments, the server may obtain the second state vector that the virtual character needs to correspond to at the second moment by: obtaining the control command triggered for the virtual character; adding the control command vector corresponding to the control command to the first state vector, and determining the result of the addition as the second state vector that the virtual character needs to correspond to at the second moment.

[0046] For example, taking a skeletal-based virtual character A in a UE4 client as an example, the server can first obtain the first state vector corresponding to virtual character A at the first moment (i.e., the vector corresponding to the first state of virtual character A at the first moment, for example, a multi-dimensional vector q can be used to represent the first state of virtual character A at the first moment, where the first state includes the first position, first orientation, and first body posture of virtual character A at the first moment). Based on the first state vector and the control commands triggered by the real player or agent for virtual character A, the server determines the second state vector that the virtual character needs to correspond to at the second moment (i.e., the vector corresponding to the second state that virtual character A expects to reach at the second moment, for example, a multi-dimensional vector g can be used to represent the second state of virtual character A at the second moment, where the second state includes the virtual character's position, first orientation, and first body posture). The second position, second orientation, and second body posture that character A needs to reach at the second moment (hereinafter referred to as the second state vector as the target) can be determined by adding the control command vector (control command vector b is a vector with the same dimension as vector a) to the first state vector. This involves adding the components of each dimension of the control command vector b to the corresponding components of the dimension in vector a. For example, assuming the first state vector a = [q1, q2, q3] and the control command vector b = [b1, b2, b3], then the vector c obtained after adding vectors q and b is [q1+b1, q2+b2, q2+b2]. In other words, the dimension of the vector will not change after adding the two vectors, but the magnitude of each component will change. The result of this addition is then determined as the second state vector that the virtual character A needs to correspond to at the second moment.

[0047] It should be noted that in the training method of the neural network model provided in this application embodiment, the first neural network model is called multiple times. Therefore, the first moment and the second moment here refer to the two moments before and after each call to the first neural network model, and not to a specific fixed moment (i.e., for each call to the first neural network model for action prediction processing, the moment before the call is called the first moment, and the moment after the call is called the second moment). The same applies to the first state vector and the second state vector. For example, taking the nth call to the first neural network model for action prediction processing (where n is an integer greater than 1) as an example, the second state vector can become the first state vector when the first neural network model is called for the (n+1)th time. For the first call to the first neural network model, the first state vector can be the vector corresponding to the initial state of the virtual character when the virtual scene is initialized or when the progress is restored from the last saved state.

[0048] In step 102, based on the first state vector and the second state vector, the first neural network model is invoked to perform action prediction processing to obtain the first action instruction, and the action control model is invoked to perform action prediction processing to obtain the second action instruction.

[0049] Here, the computational complexity of the first neural network model is less than that of the action control model.

[0050] In some embodiments, the motion control model can be a rule model that includes motion control rules (e.g., positive dynamic equations), such as a WBC controller or an MPC controller. Taking the WBC controller as an example, in order to achieve high dynamic computational accuracy, it is necessary to consider higher-order terms of Taylor expansion when calculating the second motion command, or to increase the number of iterations when solving differential equations using iterative methods. This results in the computational load increasing positively with the number of joints of the virtual character, thus making the computational complexity of the rule model greater than that of the first neural network model.

[0051] For example, when the motion control model is a rule-based model, the server can achieve the above-mentioned call to the motion control model for motion prediction processing to obtain the second motion instruction in the following way: call the rule-based model (such as WBC controller, MPC controller, etc.) to determine the second motion instruction based on the motion control rules. For example, taking WBC controller as the rule-based model, the first state vector and the second state vector can be substituted into the positive dynamic equations included in WBC controller for calculation to obtain the second motion instruction.

[0052] In other embodiments, the action control model can also be a second neural network model (e.g., a multilayer perceptron, gradient boosting tree model, support vector machine, convolutional neural network model, etc.). The complexity of the network structure of the first neural network model (e.g., the number of layers, the number of neurons) is less than that of the second neural network model. For example, the second neural network model may have more layers than the first neural network model; or, the second neural network model may have more neurons than the first neural network model. Taking the number of layers as an example, suppose the second neural network model has 100 layers while the first neural network model has only 20 layers, resulting in a higher computational complexity for the second neural network model compared to the first neural network model.

[0053] In other embodiments, see Figure 4 , Figure 4 This is a flowchart illustrating the training method for the neural network model provided in this application embodiment, as shown below. Figure 4 As shown, Figure 3 Step 102 shown can be achieved through Figure 4The implementation of steps 1021 and 1022 shown will be combined with Figure 4 The steps shown are explained.

[0054] In step 1021, the first state vector and the second state vector are concatenated, and the first neural network model is called to perform action prediction based on the concatenated vector to obtain the first action instruction.

[0055] Here, concatenation refers to adding the second state vector to the end of the first state vector. Assuming the first state vector q = [q1, q2, q3] and the second state vector g = [g1, g2, g3], then the concatenated vector d = [q1, q2, q3, g1, g2, g3] is obtained after concatenating the first state vector q and the second state vector g. It is evident that, unlike adding two vectors, concatenating two vectors changes the dimension of the vector, but the magnitude of each component remains unchanged.

[0056] In some embodiments, the first state vector may include at least one of the following state parameters: the first position of the virtual character at the first moment (e.g., the three-dimensional coordinate position of the virtual character in the world coordinate system), the first orientation of the virtual character at the first moment (e.g., the orientation of the virtual character can be characterized by three-dimensional attitude angles, where the three-dimensional attitude angles include yaw, roll, and pitch), and the first body posture of the virtual character at the first moment (e.g., the body posture of the virtual character can be characterized by the angles corresponding to multiple joints of the virtual character; assuming the virtual character has 18 joints, and the degrees of freedom of rotation of each joint range from 1 to 3, then the body posture of the virtual character can be represented by 32 angles); the second state vector may include at least one of the following state parameters: the second position that the virtual character needs to reach at the second moment ( Assuming the 3D coordinates of the virtual character's first position at the first moment are [x1, y1, z1], and the components of the control command vector corresponding to the 3D coordinates are [0, 1, 0], then the 3D coordinates of the second position the virtual character needs to reach at the second moment are [x1, y1+1, z1], the 3D coordinates of the second position the virtual character needs to reach at the second moment are [x1, y1+1, z1], the 3D orientation the virtual character needs to reach at the second moment is [R1, P1, Y1], and the components of the control command vector corresponding to the 3D orientation are [10, 5, 30], then the 3D orientation of the second orientation the virtual character needs to reach at the second moment is [R1+10, P1+5, Y1+30], and the 3D body posture the virtual character needs to reach at the second moment is [d1, d2, ..., d...]. 32The components corresponding to the 32 angles in the control command vector are [5, 10, ..., 8]. Therefore, the 32 angles corresponding to the second body posture that the virtual character needs to achieve at the second moment are [d1+5, d2+10, ..., d]. 32 +8]); then the server can concatenate the first state vector and the second state vector in the following way: determine the first difference between the first three-dimensional coordinates corresponding to the first position and the second three-dimensional coordinates corresponding to the second position; determine the second difference between the first yaw angle corresponding to the first orientation and the second yaw angle corresponding to the second orientation; concatenate the first difference, the second difference, the first roll angle and the first pitch angle corresponding to the first orientation, the second roll angle and the second pitch angle corresponding to the second orientation, the first body attitude, and the second body attitude to obtain the concatenated vector.

[0057] For example, taking virtual character A as an example, since the first neural network model focuses on the difference between the first state vector and the second state vector, rather than the first and second state vectors themselves, after the server obtains the first state vector corresponding to virtual character A at the first moment and the second state vector that virtual character A needs to correspond to at the second moment, it can first calculate the first difference (i.e., x2-x1, y2-y1, z2) between the first three-dimensional coordinates corresponding to the first position included in the first state vector (e.g., the 3D position coordinates of virtual character A in the world coordinate system at the first moment, assumed to be x1, y1, z1) and the second three-dimensional coordinates corresponding to the second position included in the second state vector (e.g., the 3D position coordinates that virtual character A needs to reach in the world coordinate system at the second moment, assumed to be x2, y2, z2). Next, the server can also calculate the second difference (i.e., Yaw2-Yaw1) between the first yaw angle corresponding to the first orientation included in the first state vector (e.g., the yaw angle corresponding to the first orientation of the virtual character A at the first moment, assuming it is Yaw1) and the second yaw angle corresponding to the second orientation included in the second state vector (e.g., the yaw angle corresponding to the second orientation of the virtual character A at the second moment, assuming it is Yaw2). Subsequently, the server can concatenate the first difference, the second difference, the first roll angle and the first pitch angle corresponding to the first orientation, the second roll angle and the second pitch angle corresponding to the second orientation, the first body posture, and the second body posture to obtain a concatenated vector. The concatenated vector is then used as the input of the first neural network model. In this way, the dimensionality of the input concatenated vector can be reduced, thereby improving training efficiency.

[0058] It should be noted that the server can also directly concatenate the first state vector and the second state vector, and use the concatenated vector as the input of the first neural network model. This application embodiment does not specifically limit this.

[0059] In step 1022, based on the first state vector and the second state vector, the motion control model is invoked to perform motion prediction processing to obtain the second motion command.

[0060] In some embodiments, when the motion control model is a rule-based model, the rule-based model may include motion control rules, and the motion control rules may be characterized by positive dynamic equations. The positive dynamic equations may include the following factors: inertia matrix, second-order time derivative of the first state vector, inertial force vector, gravity vector, and command matrix, wherein the command matrix includes internal torque commands and external torque commands. Then, the server can implement step 1022 as follows: Based on the first state vector (assuming it is denoted as q), determine the inertia matrix and gravity vector (for example, the first state vector can be substituted into the inertia equation and gravity equation respectively for calculation to obtain the corresponding inertia matrix and gravity vector); based on the first state vector and the first-order time derivative of the first state vector (i.e., angular velocity, assuming it is denoted as q), determine the inertia matrix and gravity vector (e.g., the first state vector can be substituted into the inertia equation and gravity equation respectively for calculation to obtain the corresponding inertia matrix and gravity vector); based on the first state vector and the first-order time derivative of the first state vector (i.e., angular velocity, assuming it is denoted as q), determine the inertia matrix and gravity vector. The inertial force vector is determined by substituting the first state vector and its first time derivative into the inertial force equation to obtain the inertial force vector. Based on the first state vector, the second state vector (let's call it g), the inertial force vector, the gravity vector, the inertial matrix, and the external torque command, the second time derivative of the first state vector (i.e., angular acceleration, let's call it g) is determined. Substituting the second-order time derivative of the first state vector into the positive dynamic equation, the internal torque command is obtained, and the internal torque command is determined as the second action command.

[0061] For example, the server can implement the above-mentioned determination of the second-order time derivative of the first state vector based on the first state vector, second state vector, inertial force vector, gravity vector, inertia matrix, and external torque command in the following way: determine the time difference between the second time moment and the first time moment (let's assume it's denoted as ). ), and determine the first product of the first time derivative of the first state vector and the time difference (i.e. The second state vector is subtracted from the first state vector to obtain the first subtraction result (gq), and the first subtraction result is subtracted from the first multiplication result to obtain the second subtraction result (gq-). ); the second subtraction result and the first parameter (denoted as ); Multiply the result by assuming it is a constant of 10000 to obtain the second multiplication result (i.e., ), and the result of the second multiplication is compared with the inertial force vector (let's assume it's denoted as ). Subtracting the two results yields the third subtraction result (i.e.) - ); the result of the third subtraction and the gravity vector (assuming it is) Subtracting the two results yields the fourth subtraction result (i.e.) - - ), and compare the fourth subtraction result with the external torque command (assuming it is) The two numbers are added together to obtain the first sum (i.e., ...). ); for the second parameter (denoted as (Assuming it is a constant 2000) is multiplied by the time difference to obtain the third multiplication result (i.e. ), and compare the third multiplication result with the inertia matrix (assumed to be ). The two numbers are added together to obtain the second result (i.e., ...). ); determine the inverse of the second addition result (i.e. ), and multiply the inverse by the first addition result to obtain the fourth multiplication result (i.e. The result of the fourth multiplication is determined as the second-order time derivative of the first state vector.

[0062] In step 103, a target action instruction for controlling the virtual character is selected from the first action instruction and the second action instruction based on the selection probability.

[0063] In some embodiments, after the server calls the first neural network model to perform motion prediction processing to obtain a first motion instruction and calls the motion control model to perform motion prediction processing to obtain a second motion instruction, it can send the first motion instruction and the second motion instruction to the sampler. The sampler can then select a target motion instruction for controlling the virtual character from the first motion instruction and the second motion instruction based on the selection probability. Subsequently, the sampler can send the selected target motion instruction to the interface of the physics simulation engine, so that the physics simulation engine can control the virtual character's motion based on the target motion instruction. For example, the physics simulation engine can control the corresponding joints of the virtual character to rotate to a corresponding degree based on the multiple angles included in the target motion instruction.

[0064] Here, the selection probability is gradually increased during the training of the first neural network model. This is because if the selection probability is always equal to 0, the physics simulation engine will always execute the second action command output by the action control model, and the action control model will not change as training progresses. Therefore, for the same input, only the same output will be given each time, resulting in the latest collected sample being a sample that already exists in the sample queue. That is, the latest collected sample cannot change the state distribution of the existing samples in the sample queue, which will lead to the first neural network model facing the problem of overfitting, making the final first neural network model lack generalization ability. On the other hand, if the selection probability is always 1, the physics simulation engine will only execute the first action command output by the first neural network model. However, since the first action command given by the first neural network model when it is first initialized is meaningless and completely random, many meaningless samples will be stored in the sample queue, which will make the training process of the first neural network model very slow and the training efficiency extremely low. Therefore, this embodiment of the application controls the selection probability to gradually increase during the training of the first neural network model, which can improve the training efficiency of the first neural network model on the one hand, and enhance the generalization ability of the first neural network model on the other hand.

[0065] For example, after receiving the first action command output by the first neural network model and the second action command output by the action control model, the server can simultaneously send the first and second action commands to the sampler. The sampler will then... The probability of selecting the first action instruction is given. The probability of selecting the second action instruction, and this parameter It will gradually increase from 0 to 1 as the first neural network model is trained.

[0066] In some embodiments, see Figure 5 , Figure 5 This is a flowchart illustrating the training method for the neural network model provided in this application embodiment, as shown below. Figure 5 As shown, during execution Figure 3 Before step 103 shown, the following steps can also be performed: Figure 5 Steps 106 and 107 shown will be combined Figure 5 The steps shown are explained.

[0067] In step 106, the number of samples stored in the sample queue is used as the independent variable x to construct the input variable of the function.

[0068] Here, the range of the dependent variable y of the function satisfies 0 < y < 1, and y monotonically increases as the input variable increases. That is, the function only needs to satisfy the following: its domain is in the real number R, its range is between 0 and 1, and it strictly monotonically increases as the input variable increases. For example, the function here can be a hyperbolic tangent function, a hyperbolic sine function, etc.

[0069] In step 107, the number of samples is substituted into the function, and the resulting output value is determined as the selection probability.

[0070] In some embodiments, following the above, the number of samples stored in the sample queue can be substituted into the function as the independent variable x, and the output value of the function can be determined as the selection probability. Since the number of samples stored in the sample queue is small in the early stage of training the first neural network model, the output value obtained is also small (i.e., the selection probability is small at the beginning). (The value of the selected sample is relatively small); as the training process of the first neural network model continues, the number of samples stored in the sample queue gradually increases, and the corresponding output value also gradually increases (that is, as the training continues, the probability of selection increases). (the value of which will continue to increase), thus, by selecting the probability... Initially set to 0 to improve sample efficiency, then gradually increase during training to enhance generalization.

[0071] In other embodiments, after the server selects the target instruction for controlling the virtual character from the first action instruction and the second action instruction based on the selection probability, it may also perform the following processing: multiplying the first time derivative of the first state vector with the second parameter to obtain a fifth multiplication result, and adding the fifth multiplication result with the target action instruction to obtain a third addition result; dividing the third addition result with the first parameter to obtain a first division result; adding the first division result with the first state vector to obtain a fourth addition result, and determining the fourth addition result as the reference state vector; inputting the reference state vector into the interface of the physics simulation engine so that the physics simulation engine controls the virtual character's actions based on the reference state vector.

[0072] For example, using Unreal Engine 4 as an example, the interface of the PhysicsX physics simulation engine built into the UE4 editor can only receive a reference state vector (let's call it UE4). Since the sampler cannot receive the target action command, after selecting the target action command for controlling the virtual character from the first and second action commands, the server can also perform the following processing: First, multiply the first time derivative of the first state vector with the second parameter to obtain the fifth multiplication result (i.e., ), and compare the fifth multiplication result with the target action command (let's assume it's denoted as ). The results are then added together to obtain the third sum (i.e.) + Next, the third sum is divided by the first parameter to obtain the first division result (i.e., Then, the first division result is added to the first state vector to obtain the fourth addition result (i.e. The fourth sum is then used as the reference state vector. Finally, the reference state vector is input into the interface of the physics simulation engine to complete the desired instruction settings. At this point, the physics simulation engine controls the virtual character's actions based on the reference state vector (for example, the physics simulation engine can control the corresponding joints of the virtual character to rotate according to the angles included in the reference state vector).

[0073] In step 104, the first state vector, the second state vector, and the second action instruction are combined into a sample, and the sample is stored in the sample queue.

[0074] In some embodiments, the server may store samples into a sample queue in the following manner: in response to the number of samples stored in the sample queue reaching a first quantity threshold (e.g., 10). 7 The oldest sample is removed from the sample queue in a first-in-first-out (FIFO) order, and the newest sample is stored.

[0075] For example, when the number of samples stored in the sample queue exceeds a first quantity threshold, such as when the number of samples currently stored in the sample queue exceeds 10... 7 The server can use a first-in, first-out (FIFO) approach, removing the oldest samples (those added earliest) from the sample queue and adding the most recently generated samples. Since samples added earlier contribute less to the gradient calculation of the subsequent first neural network model, and newly added samples contribute more, given limited storage space, older samples with lower gradient contributions will inevitably be removed from the sample queue.

[0076] In step 105, the first neural network model is trained based on multiple samples in the sample queue.

[0077] In some embodiments, the server may implement step 105 as follows: whenever the number of samples stored in the sample queue reaches a second quantity threshold (e.g., 10...). 5 When the input is the same, the following processing is performed: multiple samples are randomly selected from the sample queue; the first neural network model is trained based on the multiple samples until the difference between the first action command output by the first neural network model and the second action command output by the action control model is less than the difference threshold for the same input.

[0078] For example, taking a multi-layer perceptron (MLP) as the first neural network model, the MLP includes multiple layers, and each layer includes a corresponding weight matrix and bias vector. The server can implement the above-mentioned training of the first neural network model based on multiple samples in the following way: For each sample, perform the following processing: determine the difference between the second action instruction and the first action instruction included in the sample, and perform backpropagation based on the difference to update the weight matrix and bias vector corresponding to each layer in the MLP (for example, after determining the difference between the second action instruction and the first action instruction included in the sample, the gradient of the first neural network model can be determined based on the difference, and then the weight matrix and bias vector corresponding to each layer in the MLP can be updated by gradient descent).

[0079] It should be noted that, in addition to the multilayer perceptron mentioned above, the first neural network model can also be a decision tree model, gradient boosting tree model, support vector machine, convolutional neural network model, deep convolutional neural network model, phase-function neural network (PFNN), adaptive neural network (MANN), neural state machine (NSM), or fully connected neural network model, etc. The embodiments of this application do not specifically limit the type of the first neural network model.

[0080] In other embodiments, the first image frame corresponding to the first moment and the second image frame corresponding to the second moment can be two adjacent frames in the virtual scene (for example, the first image frame corresponding to the first moment is the third frame in the virtual scene, and the second image frame corresponding to the second moment is the fourth frame in the virtual scene); or, the first image frame corresponding to the first moment and the second image frame corresponding to the second moment can be two non-adjacent frames in the virtual scene (for example, assuming the first image frame corresponding to the first moment is the third frame in the virtual scene, and the second image frame corresponding to the second moment is the fifth frame in the virtual scene), and the third state vector that the virtual character needs to correspond to at the third moment corresponding to the third image frame (for example, the fourth frame in the virtual scene) can be determined by interpolation (for example, taking linear interpolation as an example, the first state vector and the second state vector can be added first, and the result of the addition divided by 2 can be determined as the third state vector), wherein the third image frame is located between the first image frame and the second image frame.

[0081] The training process of the first neural network model has now been described. The following section will continue to illustrate the role control method of the neural network model provided in this application embodiment, using exemplary applications and implementations of the terminal devices provided in this embodiment.

[0082] See Figure 6 , Figure 6 This is a flowchart illustrating the role control method for a neural network model provided in this application embodiment, which will be combined with... Figure 6 The steps shown are explained.

[0083] In step 201, the fourth state vector of the virtual character in the virtual scene at the fourth moment and the control command triggered on the virtual character are obtained.

[0084] In some embodiments, the terminal device may first obtain the fourth state vector corresponding to the virtual character in the virtual scene at the fourth moment, as well as the control instructions triggered for the virtual character. The control instructions may be triggered by a real player or by an intelligent agent (such as a robot used to simulate a real player). This application embodiment does not limit this.

[0085] In step 202, based on the control instruction vector corresponding to the control instruction and the fourth state vector, the fifth state vector that the virtual character needs to correspond to at the fifth moment is determined.

[0086] In some embodiments, after receiving a control command triggered by a user or intelligent agent for a virtual character, the terminal device can add the control command vector corresponding to the control command to the fourth state vector, and determine the addition result as the fifth state vector that the virtual character needs to correspond to at the fifth moment.

[0087] It should be noted that the fourth and fifth time points merely represent two moments before and after the process of calling the first neural network model for action prediction, and do not refer to a fixed moment. The same applies to the fourth and fifth state vectors. For example, taking the nth call to the first neural network model for action prediction (where n is an integer greater than 1) as an example, the fifth state vector can become the fourth state vector when the first neural network model is called for the (n+1)th time.

[0088] In step 203, based on the fourth and fifth state vectors, the first neural network model is invoked to perform action prediction processing to obtain the third action instruction.

[0089] Here, the first neural network model can be trained using the neural network model training method provided in the embodiments of this application.

[0090] In some embodiments, after the server trains the first neural network model based on multiple samples stored in the sample queue, it can send the trained first neural network model to the terminal device. In this way, after the terminal device obtains the fourth state vector and the fifth state vector, it can first concatenate the fourth state vector and the fifth state vector, and use the concatenated vector as the input of the first neural network model, so that the first neural network model can perform action prediction processing based on the concatenated vector and output the third action command.

[0091] In step 204, the actions of the virtual character are controlled according to the third action instruction.

[0092] In some embodiments, after receiving the third action instruction, the terminal device can input the third action instruction to the interface of the physics simulation engine so that the physics simulation engine can control the action of the virtual character according to the third action instruction. For example, if the third action instruction carries the angle values ​​that each joint of the virtual character needs to rotate, the physics simulation engine can control the corresponding joint of the virtual character to rotate according to the angle values.

[0093] The neural network model training method provided in this application continuously increases the probability of selecting the first action command from the first action command output by the first neural network model and the second action command output by the action control model during the training process of the first neural network model. This enables the first neural network model with low computational complexity to continuously fit the control capability of the action control model with high computational complexity. Thus, when the first neural network model replaces the action control model to control the action of the virtual character, it can achieve precise control of the virtual character in the virtual scene with the lowest possible computational resources and time, thereby obtaining smooth, realistic, and high-quality animation.

[0094] Furthermore, it should be noted that since the motion control model has already considered the impact of characteristics such as object collisions and terrain undulations in the virtual scene on the virtual character's movements during its design, and the first neural network model is obtained by fitting the control capabilities of the motion control model, when using the first neural network model to replace the motion control model to control the virtual character's movements, it can also cope with the characteristics of object collisions and terrain undulations in the game environment, thereby obtaining realistic animation effects.

[0095] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0096] This application proposes a data-fitting-based controller method for generating skeletal animations with dynamic simulations. This method uses the Dagger algorithm and a multi-layer fully connected neural network to fit an existing rule-based dynamic simulation controller, such as a WBC controller, MPC controller, or a Stable Proportional-Derivative (SPD) controller, to control specific actions of a virtual character with dynamic simulations within a physics engine. The controller fitted using a neural network model has the advantages of fast online computation speed and insensitivity to the high dynamic characteristics of the actions. Furthermore, the neural network model itself is inherently a parameterization of control strategies, facilitating integration with deep reinforcement learning algorithms. Further adjustments to the parameters of the neural network model using deep reinforcement learning algorithms can achieve more precise control results than rule-based controllers.

[0097] The following specific example illustrates the solution provided in this application. For instance, a fully connected neural network model can be used to fit the SPD controller to generate real-time controlled animations of characters with dynamics. It should be noted that this SPD controller can be replaced with any other existing complex controller, such as a WBC controller or an MPC controller. Of course, a small-sized policy neural network can also be used to fit a known large-sized neural network; this application does not limit this approach.

[0098] The solution provided in this application includes two parts: training the neural network model and testing the neural network model. The training process of the neural network model will be described first below.

[0099] For example, see Figure 7 , Figure 7 This is a schematic diagram of the training process of the neural network model provided in the embodiments of this application, as shown below. Figure 7 As shown, in each frame of the game, the character's current state q (corresponding to the first state vector mentioned above) and the target g obtained from the player's real-time input (corresponding to the second state vector mentioned above) constitute the input of the policy neural network (corresponding to the first neural network model mentioned above). The policy neural network outputs an instruction a (corresponding to the first action instruction mentioned above). At the same time, state q and target g also become the input of the rule controller (corresponding to the action control model mentioned above, here the SPD controller is used), which provides an optimal instruction. In the sampler, there are There is a probability that instruction a will be selected, and a probability of (1-a) that the best instruction will be selected. (Corresponding to the second action command mentioned above), and sent to the dynamics engine to calculate the next frame. This parameter It will gradually increase from 0 to 1 as training progresses. Furthermore, regardless of which instruction is selected for the sampler, it will only sample (q, g, ... Each sample is saved as a sample in the experience replay pool. This experience replay pool can be implemented using a queue, which removes the earliest batch of samples every certain period (e.g., every 30 minutes). Whenever a certain number of samples are collected, the gradient descent optimizer samples a batch of samples from the experience replay pool and updates the parameters of the policy neural network using gradient descent. After a certain number of iterations, the policy neural network can fit the rule controller.

[0100] The testing process for the neural network model will be explained below.

[0101] For example, see Figure 8 , Figure 8 This is a schematic diagram of the testing process of the neural network model provided in the embodiments of this application, as shown below. Figure 8 As shown, after the policy neural network completes training, the rule controller, sampler, and other network training-related modules can be completely removed from the program. During gameplay, each frame is executed solely by the policy neural network, which quickly calculates the dynamics engine's instruction a based on the character's current state q and the target g, to meet the requirements for real-time generation of high-quality animation.

[0102] The method used in this application embodiment for finding a control strategy for animation generation of virtual characters based on dynamic simulation is essentially a supervised learning method. The entire process of acquiring the policy neural network is as follows: Figure 9 As shown, the following will be combined with Figure 9 The steps shown are explained.

[0103] In step 301, the policy neural network is initialized.

[0104] This application uses a neural network model to fit the control strategy given by the rule controller; therefore, the neural network model can be called a policy neural network. Initially, a control strategy is obtained by randomly initializing the parameters of the policy neural network.

[0105] In step 302, samples are collected.

[0106] In some embodiments, such as Figure 7 As shown, samples can be collected during continuous interaction with the bipedal skeleton character in the UE4 client.

[0107] In step 303, the policy neural network is trained.

[0108] In some embodiments, after a sample enters the experience replay pool, policy gradient estimation and parameter upgrades of the policy neural network can be performed based on the samples stored in the experience replay pool.

[0109] In step 304, the sampler coefficients are adjusted.

[0110] In some embodiments, during the training of the policy neural network, the selection probability of different instructions in the sampler can be adjusted.

[0111] In step 305, it is determined whether the control effect meets the standard. If it does not meet the standard, step 302 is executed; if it meets the standard, step 306 is executed.

[0112] In some embodiments, the server can determine whether the control effect of the policy neural network meets the target (i.e., determine whether the instruction 'a' output by the policy neural network matches the optimal instruction output by the rule controller). If the difference between the samples is less than the difference threshold, and the threshold is not met, then proceed to step 302 to obtain new samples for the next round of training.

[0113] In step 306, a policy neural network is used instead of a rule controller.

[0114] In some embodiments, when the control effect of the policy neural network meets the requirements, the rule controller can be removed in the UE4 client and the policy neural network can be used to replace the rule controller to control the actions of the virtual character.

[0115] The following will address... Figure 7 The various sub-modules shown include state q, target g, instruction a, and optimal instruction. The meaning of SPD, the mathematical expression of policy neural network, sampler, experience replay pool, and the relationship between instruction and dynamic system are explained in detail.

[0116] The following section will first discuss the state q, the target g, and the instruction a and the optimal instruction. The meaning of will be explained.

[0117] In some embodiments, state q describes the position, orientation, and body posture of a skeleton-based virtual character at a given moment. For example, see [example missing]. Figure 10 , Figure 10 This is a schematic diagram of a bipedal character model in Unreal Engine 4 provided in an embodiment of this application. Figure 10This demonstrates the relationship between the component coordinate system (1002) and the pelvis coordinate system (1001) of a bipedal character model in the UE4 client. The pelvis is the root of the entire character's skeleton; its position and pose in the world coordinate system represent the character's position and orientation within that system. It has six degrees of freedom, which can be represented by six quantities. Three of these quantities are three-dimensional position coordinates, and the other three are three angles representing orientation: roll, pitch, and yaw.

[0118] Furthermore, the body posture of a skeletal character can be represented using 32 angles. For example... Figure 10 As shown, in the example bipedal human skeleton model, besides Pelvis as the root joint, there are 18 other rotatable joints listed in Table 1. The degrees of freedom for each joint range from 1 to 3. Therefore, the 18 joints generate a total of 32 degrees of freedom. Thus, using the 18 sets of quaternions representing bone rotation in the data package, 32 angle values ​​can be calculated. Using these 32 angle values, plus Pelvis's 3D position coordinates and 3D pose angles in the world coordinate system, a total of 38-dimensional vectors can fully represent the character's current state. Therefore, state q is this 38-dimensional vector.

[0119] Table 1 Rotatable joints and their degrees of freedom

[0120] It should be noted that, Figure 7 , 8 Table 1 shows the image of a bipedal character model in the UE4 engine, while the solution provided in this application embodiment can be used for any animated character model based on tandem skeletons. Moreover, the more supporting ends (e.g., feet) a skeleton model has, the simpler the corresponding motion control rules become, and thus the easier it is to obtain a control strategy.

[0121] In some embodiments, the target g can also be a 38-dimensional vector, with each component having the same meaning as the state q. However, the target g is not the actual state the character reaches, but rather the state the character expects to reach in the next moment, calculated from the character's current state q and the player's input. Figure 7 The user input processing module can be handled by either PFNN or NSM. For example... Figure 7 The training process can be fully automated, meaning it can use the previously trained PFNN or NSM to simulate the input information of a real player by arranging and combining the input signals in the dataset. Of course, input can also be provided by a real player; this application does not specifically limit this approach.

[0122] In some embodiments, a dynamic simulator is equivalent to... Figure 7 The skeletal character shown has 32 rotating motors mounted on its body, corresponding to the 32 joint angles in Table 1. At each moment, torque commands need to be supplied to the motors. The dynamics simulator will adjust the force based on the character's current state (q) and torque commands. The system calculates the skeletal character's state in the next moment by considering factors such as gravity and external support forces. Therefore, in Figure 7 Whether it's the instruction 'a' output by the policy neural network or the optimal instruction output by the rule controller... Each of them can be represented by a 32-dimensional vector.

[0123] The following continues... Figure 7 The SPD-based rule controller shown in the figure is explained.

[0124] In some embodiments, the role of the rule controller is to find a suitable instruction based on the current state q of the role and the target g. The goal is to make the state simulated by the dynamic simulator at the next moment as close as possible to the target g. The SPD controller provides a way to find the optimal instruction. The method.

[0125] For example, the goal is to deduce the appropriate instructions from the expected target g to be reached in the next moment. Therefore, it is necessary to know the torque command of the motor in the dynamic simulator. How is the state at the next moment simulated and calculated? This process can be described by the following positive dynamic equations of the skeletal model: (1) in, and These are the first and second time derivatives of state q, which are the (angular) velocity and (angular) acceleration, respectively. Inertia matrix. It is The dimensional matrix is ​​calculated from the current state (i.e., the first state mentioned above) q. Inertial force vector. It is a 38-dimensional vector that can be represented by q and Calculated. Gravity vector. This represents the effect of gravitational acceleration on each joint of the skeleton, and is also a 38-dimensional vector, calculated from q. The right-hand side of the equation represents the internal torque generated by the motor. The external torque generated at each joint angle by the external forces (such as support force and friction) acting on the character. . Notice and Only 32-dimensional vectors, therefore the matrix The inclusion of only 0s and 1s is simply to add six zeros in front of the vector, thus making the right side of the equation a 38-dimensional vector, forming an equation with the left side. For simplicity, these will be automatically omitted in the following derivation. Therefore, when the vector dimensions on both sides of an equation are misaligned, six zeros are automatically added before the 32-dimensional vector to augment it into a 38-dimensional vector.

[0126] Therefore, dynamic simulators are all based on the positive dynamic equations, using the known q, , , Calculate Then through , , We perform a double integral to obtain the state at the next moment. Then we hope this state will approach the objective. Inspired by traditional PD controllers, internal torque commands typically take the following form: (2) in, Represents the time difference between the next moment and the current moment, parameter It can be specified as a constant 10000, parameter It can be specified as a constant of 2000. Therefore, substituting the above equation into the positive dynamic equation eliminates... You will get: (3) Again Substituting the expression for the known quantity into the initial positive dynamic equation, we can deduce the result. Here it is, this is the optimal instruction given by the SPD-based rule controller. This also corresponds to Figure 7 The expression given in [the document]. In practice, this optimal instruction [is used]. Although it cannot perfectly track the state to the target g in the next moment, the SPD controller obviously has a much better control effect than a simple PD controller.

[0127] Furthermore, it should be noted that the reason for not directly inserting an SPD controller to generate skeletal character animations containing dynamic simulations during gameplay is because the inertia matrix in the aforementioned positive dynamic equations... Inertial force vector The computational load is extremely high. As the number of joints in a skeletal character increases, iterative methods become necessary to solve the problem. and The computing power requirements have skyrocketed, making it completely impossible to meet the real-time rendering needs of the client used by players.

[0128] Therefore, the solution provided in this application embodiment will fit the SPD controller through a policy neural network. Thus, as... Figure 8 As shown, there is no need to deploy an SPD controller for computation on the client used by the player. The instructions of the dynamic simulator can be calculated quickly and in real time through simple neural network forward computation.

[0129] The following continues... Figure 7 The policy neural network shown in the figure will be explained.

[0130] In some embodiments, such as Figure 7 As shown, the 38-dimensional state q and the 38-dimensional target g will constitute the input of the 72-dimensional policy neural network. The dimension of the vector is reduced because the four quantities in the state q and target g—the 3-dimensional position coordinates of Pelvis and the yaw attitude angle—do not directly constitute the input. Instead, they are represented by the pairwise differences between them (because the policy neural network is most interested in the differences between the state q and the target g, not the state q and the target g themselves). Therefore, the input has four fewer components, which improves the sample efficiency of the algorithm.

[0131] For example, taking a policy neural network as an MLP, let's assume... Figure 7 The policy neural network shown is a 3-layer perceptron, and its expression can be written as: Output = W2 tanh(W1 tanh(W0 Input + b0) + b1) + b2 (4) In this context, matrices W0, W1, W2, and bias vectors b0, b1, b2 are collectively referred to as the weights of the policy neural network. The network's output is... Figure 7The instruction 'a' in the expression is a 32-dimensional vector used to control the rotation of the virtual character's joints. As can be seen from this expression, the policy neural network has two hidden layers. The number of neurons in each hidden layer can be selected according to the complexity of the virtual character's animation (i.e., the more complex the action the virtual character needs to perform, the more neurons are needed in each hidden layer; the simpler the action, the fewer neurons are needed in each hidden layer). For example, in this embodiment, if the character is to perform simple walking and running tasks, only a small number of neurons (e.g., 128) are needed in each hidden layer. However, when the character is to perform complex actions (e.g., martial arts, professional sports movements), a large number of neurons (e.g., 512) are needed in each hidden layer.

[0132] The following continues... Figure 7 The sampler and experience playback pool shown in the figure are explained.

[0133] In some embodiments, such as Figure 7 As shown, at any given time, the sampler will simultaneously receive the optimal instruction output by the rule controller. The policy neural network gives instruction 'a'. The sampler will perform random sampling, choosing one of two options with probability. Choose instruction a, or use (1- Probability choice This is sent to the dynamics engine as the final instruction. The value will gradually increase from 0 to 1 during training. Subsequently, regardless of which instruction the sampler selects, only the best instruction given by the rule controller will be saved. The inputs to the policy neural network at this time (i.e., state q and target g) are sent to the experience replay pool.

[0134] It should be noted that the probability is controlled here. The reason why the value increases from 0 to 1 as training progresses is because: Imagine if If the value is always 0, the physics engine will always execute the best instructions given by the rule controller. Because the rule controller does not change during training, it will produce the same output for the same input each time. Over time, the most recently collected samples are all samples already seen in the experience replay pool; or, in statistical terms, the most recently collected samples cannot change the state distribution of existing samples in the experience replay pool. This will lead to overfitting in the training of the policy neural network, making the final policy neural network completely lacking in generalization ability and unable to perform well in training. Figure 8 It works normally in the test environment shown. That's the assumption. The value is always 1, meaning the physics engine only executes the instruction 'a' output by the policy neural network. Since the calculation result given by the policy neural network when it is first initialized is meaningless and completely random, there will be many meaningless samples in the experience replay pool. The states 'q' contained in these samples are basically impossible to see in the normal game process. This will cause the training process of the neural network model to be extremely slow and the training sample efficiency to be extremely low.

[0135] Therefore, in summary, the embodiments of this application require... Initially set to 0 to improve sample efficiency, then gradually increased to no more than 1 during training to enhance generalization. In some embodiments, The expression can be written as: (5) Here, x represents the number of samples collected in the experience replay pool, and tanh is the hyperbolic tangent function, defined over the real number R, with a range from -1 to 1, and strictly monotonically increasing with x. The experience replay pool utilizes the Dagger algorithm to store the collected samples in a queue data structure. For example, the maximum queue size can be 10. 7 The number of samples collected exceeds the maximum value, so the earliest sample stored in the queue is removed from the queue.

[0136] The reason for using a queue to implement the experience replay pool is that, statistically speaking, the earlier a sample enters the experience replay pool, the smaller its contribution to the gradient calculation of the current policy neural network; conversely, the sample data that has just entered the experience replay pool will contribute more to the gradient calculation of the policy neural network. Given the limited storage space, older sample data is removed from the queue.

[0137] Finally, after collecting a certain number (e.g., 10) 5 Given a sample of data (number of samples), the gradient descent optimizer will randomly sample several batches of data from the experience replay pool and perform stochastic gradient descent (SGD) operations to update the weight matrices W0, W1, W2 and bias vectors b0, b1, b2 of the policy neural network. Then, it will continue to collect sample data. This process is repeated until a satisfactory policy neural network is obtained, which can provide instructions that differ from the optimal instructions output by the rule controller by less than a difference threshold in all actions required by the game.

[0138] The relationship between commands and the dynamics engine will be explained below.

[0139] In some embodiments, the sampler compares the instruction 'a' given by the policy neural network with the optimal instruction given by the rule controller. Choose one of them as the target instruction. Input into the positive dynamic equation. After calculation... After that, through q, , The state at the next moment is obtained by performing a double integral. In this embodiment, since the game is developed using the UE4 editor, and its built-in PhysicsX physics simulation engine does not have an interface for receiving instructions... Instead, refer to the state vector. And it is calculated using the following formula: (6) Therefore, a simple transformation can be made to the above formula, according to the required... This leads to the derivation of the reference state vector that should be input to the interface. : (7) This reference state vector The desired command settings can be completed by inputting them into the PhysicsX physics simulation engine interface.

[0140] This application proposes an artificial intelligence method that utilizes the Dagger algorithm to fit a complex controller in a skeletal-based game character dynamics simulation using a neural network model. This aims to generate control strategies for characters in UE4 with physics-based simulation enabled. The control strategies obtained using this approach can generate smooth, realistic, and high-quality animations with environmental collision interactions in real-time for game characters in UE4 with lower computational resources and time. Furthermore, this parameterized control strategy can be easily integrated with deep reinforcement learning algorithms to improve animation quality.

[0141] The following description continues to illustrate the exemplary structure of the neural network model training device 243 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2A As shown, the software modules in the training device 243 of the neural network model stored in the memory 240 may include: an acquisition module 2431, an action prediction module 2432, a selection module 2433, a combination module 2434, a storage module 2435, and a training module 2436.

[0142] The module 2431 is used to acquire the first state vector corresponding to the virtual character in the virtual scene at the first moment, and to acquire the second state vector required by the virtual character at the second moment; the action prediction module 2432 is used to call the first neural network model to perform action prediction processing to obtain the first action command based on the first state vector and the second state vector, and to call the action control model to perform action prediction processing to obtain the second action command, wherein the computational complexity of the first neural network model is less than that of the action control model; the selection module 2433 is used to select the target action command for controlling the virtual character from the first action command and the second action command based on the selection probability, wherein the selection probability gradually increases during the training process of the first neural network model; the combination module 2434 is used to combine the first state vector, the second state vector, and the second action command into a sample; the storage module 2435 is used to store the sample in a sample queue; the training module 2436 is used to train the first neural network model based on multiple samples in the sample queue, wherein the first neural network model is used to replace the action control model to control the action of the virtual character.

[0143] In some embodiments, the type of action control model includes: a rule model including action control rules, and a second neural network model; the action prediction module 2432 is further configured to call the rule model to determine a second action instruction based on the action control rules; or, call the trained second neural network model to perform action prediction processing to obtain a second action instruction, wherein the complexity of the network structure of the first neural network model is less than the complexity of the network structure of the second neural network model.

[0144] In some embodiments, the acquisition module 2431 is further configured to acquire control commands triggered for the virtual character; the training device 243 of the neural network model further includes a determination module 2437, configured to add the control command vector corresponding to the control command to the first state vector, and determine the addition result as the second state vector that the virtual character needs to correspond to at the second moment.

[0145] In some embodiments, the training device 243 of the neural network model further includes a splicing module 2438 for splicing the first state vector and the second state vector; an action prediction module 2432 for calling the first neural network model to perform action prediction processing based on the spliced ​​vector to obtain a first action instruction; and for calling the action control model to perform action prediction processing based on the first state vector and the second state vector to obtain a second action instruction.

[0146] In some embodiments, the first state vector includes at least one of the following state parameters: the first position of the virtual character at the first moment, the first orientation of the virtual character at the first moment, and the first body posture of the virtual character at the first moment; the second state vector includes at least one of the following state parameters: the second position that the virtual character needs to reach at the second moment, the second orientation that the virtual character needs to reach at the second moment, and the second body posture that the virtual character needs to reach at the second moment; the determining module 2437 is further configured to determine a first difference between the first three-dimensional coordinates corresponding to the first position and the second three-dimensional coordinates corresponding to the second position; and to determine a second difference between the first yaw angle corresponding to the first orientation and the second yaw angle corresponding to the second orientation; the stitching module 2438 is further configured to stitch the first difference, the second difference, the first roll angle and the first pitch angle corresponding to the first orientation, the second roll angle and the second pitch angle corresponding to the second orientation, the first body posture, and the second body posture to obtain a stitched vector.

[0147] In some embodiments, the motion control model includes a rule model, which includes motion control rules characterized by a positive dynamic equation. The positive dynamic equation includes the following factors: an inertia matrix, the second time derivative of a first state vector, an inertial force vector, a gravity vector, and a command matrix, wherein the command matrix includes an internal torque command and an external torque command. The motion prediction module 2432 is further configured to perform the following processing: determining the inertia matrix and the gravity vector based on the first state vector; determining the inertial force vector based on the first state vector and the first time derivative of the first state vector; determining the second time derivative of the first state vector based on the first state vector, the second state vector, the inertial force vector, the gravity vector, the inertia matrix, and the external torque command; substituting the second time derivative of the first state vector into the positive dynamic equation to obtain the internal torque command, and determining the internal torque command as the second motion command.

[0148] In some embodiments, the determining module 2437 is further configured to: determine the time difference between the second time moment and the first time moment; determine the first multiplication result of the first time derivative of the first state vector and the time difference; subtract the second state vector from the first state vector to obtain the first subtraction result; subtract the first subtraction result from the first multiplication result to obtain the second subtraction result; multiply the second subtraction result with the first parameter to obtain the second multiplication result; subtract the second multiplication result from the inertial force vector to obtain the third subtraction result; subtract the third subtraction result from the gravity vector to obtain the fourth subtraction result; add the fourth subtraction result to the external torque command to obtain the first addition result; multiply the second parameter with the time difference to obtain the third multiplication result; add the third multiplication result to the inertial matrix to obtain the second addition result; determine the inverse of the second addition result; multiply the inverse with the first addition result to obtain the fourth multiplication result; and determine the fourth multiplication result as the second time derivative of the first state vector.

[0149] In some embodiments, the determining module 2437 is further configured to determine the selection probability by means of the following method before selecting the target action instruction for controlling the virtual character from the first action instruction and the second action instruction based on the selection probability: using the number of samples stored in the sample queue as the independent variable x, constructing the input variable of the function, wherein the range of the dependent variable y of the function satisfies 0 < <1, and y increases monotonically as the input variable increases; substitute the number of samples into the function, and determine the output value as the selection probability.

[0150] In some embodiments, the storage module 2435 is further configured to, in response to the number of samples stored in the sample queue reaching a first quantity threshold, delete the oldest sample from the sample queue in a first-in-first-out order and store the latest generated sample.

[0151] In some embodiments, the training module 2436 is further configured to perform the following processing whenever the number of samples stored in the sample queue reaches a second quantity threshold: randomly select multiple samples from the sample queue; train a first neural network model based on the multiple samples until the difference between the first action command output by the first neural network model and the second action command output by the action control model is less than the difference threshold.

[0152] In some embodiments, the first neural network model is a multilayer perceptron, which includes multiple layers, and each layer includes a corresponding weight matrix and bias vector; the training module 2436 is further configured to perform the following processing for each sample: determine the difference between the second action instruction included in the sample and the first action instruction, and perform backpropagation processing based on the difference to update the weight matrix and bias vector corresponding to each layer in the multilayer perceptron.

[0153] In some embodiments, the first image frame corresponding to the first moment and the second image frame corresponding to the second moment are two adjacent frames in the virtual scene; or, the first image frame corresponding to the first moment and the second image frame corresponding to the second moment are two non-adjacent frames in the virtual scene, and the third state vector that the virtual character needs to correspond to at the third moment corresponding to the third image frame is determined by interpolation, and the third image frame is located between the first image frame and the second image frame.

[0154] In some embodiments, the training device 243 for the neural network model further includes a conversion module 24310 for converting the target action command into a reference state vector adapted to the physics simulation engine; the training device 243 for the neural network model further includes an input module 24311 for inputting the reference state vector into the interface of the physics simulation engine so that the physics simulation engine controls the action of the virtual character based on the reference state vector.

[0155] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment, therefore it will not be repeated. For technical details not covered in the neural network model training apparatus provided in this application embodiment, please refer to... Figures 3 to 5 The description of any of the accompanying drawings is for reference only.

[0156] The following description continues to illustrate the exemplary structure of the role control device 455 for the neural network model provided in this application embodiment as a software module. In some embodiments, such as... Figure 2B As shown, the software modules in the role control device 455 storing the neural network model in the memory 450 may include: an acquisition module 4551, a determination module 4552, an action prediction module 4553, and a control module 4554.

[0157] The acquisition module 4551 is used to acquire the fourth state vector of the virtual character in the virtual scene at the fourth moment, and the control command triggered for the virtual character; the determination module 4552 is used to determine the fifth state vector that the virtual character needs to correspond to at the fifth moment based on the control command vector corresponding to the control command and the fourth state vector; the action prediction module 4553 is used to call the first neural network model to perform action prediction processing based on the fourth state vector and the fifth state vector to obtain the third action command; wherein, the first neural network model is trained according to the neural network model training method provided in the embodiments of this application; the control module 4554 is used to control the action of the virtual character according to the third action command.

[0158] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the neural network model training method or role control method described in this application.

[0159] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute a training method or a role control method for a neural network model provided in this application. For example... Figures 3 to 5 The training method of the neural network model shown in any of the attached figures, or Figure 6 The role control method of the neural network model is shown.

[0160] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0161] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0162] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0163] As an example, executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0164] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for training a neural network model, characterized in that, The method includes: Obtain the first state vector of the virtual character in the virtual scene at the first moment, and obtain the second state vector of the virtual character at the second moment; Based on the first state vector and the second state vector, a first neural network model is invoked to perform action prediction processing to obtain a first action instruction, and an action control model is invoked to perform action prediction processing to obtain a second action instruction. The computational complexity of the first neural network model is less than that of the action control model. Based on the selection probability, a target action instruction for controlling the virtual character is selected from the first action instruction and the second action instruction, wherein the selection probability is the probability of selecting the first action instruction, the probability of selecting the second action instruction is 1 minus the selection probability, and the selection probability gradually increases during the training process of the first neural network model. The target action command is converted into a reference state vector adapted to the physics simulation engine; The reference state vector is input into the interface of the physics simulation engine so that the physics simulation engine controls the actions of the virtual character based on the reference state vector; The first state vector, the second state vector, and the second action instruction are combined into a sample, and the sample is stored in a sample queue. The first neural network model is trained based on multiple samples in the sample queue, wherein the first neural network model is used to replace the motion control model to control the actions of the virtual character.

2. The method according to claim 1, characterized in that, The types of motion control models include: rule models that include motion control rules, and second neural network models; The step of calling the motion control model to perform motion prediction processing to obtain the second motion instruction includes: Invoke the rule model to determine the second action instruction based on the action control rules; or, The trained second neural network model is invoked to perform action prediction processing to obtain a second action instruction, wherein the complexity of the network structure of the first neural network model is less than that of the network structure of the second neural network model.

3. The method according to claim 1, characterized in that, The step of obtaining the second state vector required for the virtual character at the second moment includes: Obtain the control commands triggered for the virtual character; The control command vector corresponding to the control command is added to the first state vector, and the result of the addition is determined as the second state vector that the virtual character needs to correspond to at the second moment.

4. The method according to claim 1, characterized in that, The step of calling a first neural network model to perform action prediction processing to obtain a first action command based on the first state vector and the second state vector, and calling an action control model to perform action prediction processing to obtain a second action command, includes: The first state vector and the second state vector are concatenated, and the first neural network model is called to perform action prediction based on the concatenated vector to obtain the first action command. Based on the first state vector and the second state vector, the motion control model is invoked to perform motion prediction processing to obtain the second motion command.

5. The method according to claim 4, characterized in that, The first state vector includes at least one of the following state parameters: the first position of the virtual character at the first moment, the first orientation of the virtual character at the first moment, and the first body posture of the virtual character at the first moment; The second state vector includes at least one of the following state parameters: the second position that the virtual character needs to reach at the second moment, the second orientation that the virtual character needs to achieve at the second moment, and the second body posture that the virtual character needs to achieve at the second moment; The concatenation process of the first state vector and the second state vector includes: Determine the first difference between the first three-dimensional coordinates corresponding to the first position and the second three-dimensional coordinates corresponding to the second position; Determine the second difference between the first yaw angle corresponding to the first orientation and the second yaw angle corresponding to the second orientation; The first difference, the second difference, the first roll angle and the first pitch angle corresponding to the first orientation, the second roll angle and the second pitch angle corresponding to the second orientation, the first body posture, and the second body posture are spliced ​​together to obtain a spliced ​​vector.

6. The method according to claim 4, characterized in that, The motion control model includes a rule model, which includes motion control rules. The motion control rules are characterized by a positive dynamic equation, which includes the following factors: inertia matrix, second-order time derivative of the first state vector, inertial force vector, gravity vector, and command matrix, wherein the command matrix includes internal torque command and external torque command. The step of calling the action control model to perform action prediction processing based on the first state vector and the second state vector to obtain the second action command includes: Based on the first state vector, the inertia matrix and the gravity vector are determined; The inertial force vector is determined based on the first state vector and the first time derivative of the first state vector. Based on the first state vector, the second state vector, the inertial force vector, the gravity vector, the inertial matrix, and the external torque command, the second time derivative of the first state vector is determined; Substituting the second-order time derivative of the first state vector into the positive dynamic equation yields the internal torque command, which is then determined as the second action command.

7. The method according to claim 6, characterized in that, The step of determining the second-order time derivative of the first state vector based on the first state vector, the second state vector, the inertial force vector, the gravity vector, the inertial matrix, and the external torque command includes: Determine the time difference between the second time point and the first time point, and determine the first multiplication result of the first time derivative of the first state vector and the time difference; Subtract the second state vector from the first state vector to obtain a first subtraction result, and subtract the first subtraction result from the first multiplication result to obtain a second subtraction result; The second subtraction result is multiplied by the first parameter to obtain the second multiplication result, and the second multiplication result is subtracted from the inertial force vector to obtain the third subtraction result; The third subtraction result is subtracted from the gravity vector to obtain the fourth subtraction result, and the fourth subtraction result is added to the external torque command to obtain the first addition result; The second parameter is multiplied by the time difference to obtain a third multiplication result, and the third multiplication result is added to the inertia matrix to obtain a second addition result; Determine the inverse of the second addition result, and multiply the inverse with the first addition result to obtain the fourth multiplication result; The result of the fourth multiplication is determined as the second-order time derivative of the first state vector.

8. The method according to any one of claims 1 to 7, characterized in that, Before selecting a target action command for controlling the virtual character from the first action command and the second action command based on a selection probability, the method further includes: The selection probability is determined in the following manner: The number of samples stored in the sample queue is used as the independent variable x to construct the input variable of the function, wherein the range of the dependent variable y of the function satisfies 0 < <1, and y increases monotonically as the input variable increases; Substitute the number of samples into the function, and determine the output value as the selection probability.

9. The method according to any one of claims 1 to 7, characterized in that, The step of storing the sample in the sample queue includes: In response to the number of samples stored in the sample queue reaching a first quantity threshold, the oldest sample is deleted from the sample queue in a first-in-first-out order, and the most recently generated sample is stored.

10. The method according to any one of claims 1 to 7, characterized in that, Training the first neural network model based on multiple samples in the sample queue includes: Whenever the number of samples stored in the sample queue reaches the second threshold, the following processing is performed: Randomly select multiple samples from the sample queue; The first neural network model is trained based on multiple samples until the difference between the first action command output by the first neural network model and the second action command output by the action control model is less than a difference threshold.

11. The method according to claim 10, characterized in that, The first neural network model is a multilayer perceptron, which includes multiple layers, and each layer includes a corresponding weight matrix and bias vector; The step of training the first neural network model based on multiple samples includes: For each of the samples, perform the following processing: The difference between the second action command and the first action command included in the sample is determined, and backpropagation is performed based on the difference to update the weight matrix and bias vector corresponding to each layer in the multilayer perceptron.

12. The method according to any one of claims 1 to 7, characterized in that, The first image frame corresponding to the first time point and the second image frame corresponding to the second time point are two adjacent frames in the virtual scene; or... The first image frame corresponding to the first moment and the second image frame corresponding to the second moment are two non-adjacent frames in the virtual scene, and the third state vector that the virtual character needs to correspond to at the third moment corresponding to the third image frame is determined by interpolation. The third image frame is located between the first image frame and the second image frame.

13. A method for role control in a neural network model, characterized in that, The method includes: Obtain the fourth state vector of the virtual character in the virtual scene at the fourth moment, and the control command triggered for the virtual character; Based on the control instruction vector corresponding to the control instruction and the fourth state vector, determine the fifth state vector that the virtual character needs to correspond to at the fifth moment; Based on the fourth state vector and the fifth state vector, the first neural network model is invoked to perform action prediction processing to obtain a third action instruction, and the action of the virtual character is controlled according to the third action instruction; wherein, the first neural network model is trained by the training method of the neural network model according to any one of claims 1 to 12.

14. A training device for a neural network model, characterized in that, The device includes: The acquisition module is used to acquire the first state vector corresponding to the virtual character in the virtual scene at the first moment, and to acquire the second state vector that the virtual character needs to correspond to at the second moment. The action prediction module is used to call a first neural network model to perform action prediction processing to obtain a first action instruction based on the first state vector and the second state vector, and to call an action control model to perform action prediction processing to obtain a second action instruction, wherein the computational complexity of the first neural network model is less than that of the action control model. The selection module is used to select a target action instruction for controlling the virtual character from the first action instruction and the second action instruction based on the selection probability, wherein the selection probability is the probability of selecting the first action instruction, the probability of selecting the second action instruction is 1 minus the selection probability, and the selection probability gradually increases during the training process of the first neural network model. A conversion module is used to convert the target action command into a reference state vector adapted to the physics simulation engine; An input module is used to input the reference state vector into the interface of the physics simulation engine, so that the physics simulation engine controls the action of the virtual character based on the reference state vector; The combination module is used to combine the first state vector, the second state vector, and the second action instruction into a sample; A storage module is used to store the samples into a sample queue; The training module is used to train the first neural network model based on multiple samples in the sample queue, wherein the first neural network model is used to replace the motion control model to control the actions of the virtual character.

15. The apparatus according to claim 14, characterized in that, The types of motion control models include: rule models that include motion control rules, and second neural network models; The action prediction module is further configured to invoke the rule model to determine a second action instruction based on the action control rules; or, invoke the trained second neural network model to perform action prediction processing to obtain a second action instruction, wherein the complexity of the network structure of the first neural network model is less than the complexity of the network structure of the second neural network model.

16. The apparatus according to claim 14, characterized in that, The acquisition module is also used to acquire control commands triggered on the virtual character; The device further includes: The determination module is used to add the control instruction vector corresponding to the control instruction to the first state vector, and determine the addition result as the second state vector that the virtual character needs to correspond to at the second moment.

17. The apparatus according to claim 14, characterized in that, The device further includes: The splicing module is used to splice the first state vector and the second state vector. The action prediction module is further configured to call the first neural network model to perform action prediction processing based on the spliced ​​vector obtained by splicing, and obtain a first action instruction; and to call the action control model to perform action prediction processing based on the first state vector and the second state vector, and obtain a second action instruction.

18. The apparatus according to claim 17, characterized in that, The first state vector includes at least one of the following state parameters: the first position of the virtual character at the first moment, the first orientation of the virtual character at the first moment, and the first body posture of the virtual character at the first moment; The second state vector includes at least one of the following state parameters: the second position that the virtual character needs to reach at the second moment, the second orientation that the virtual character needs to achieve at the second moment, and the second body posture that the virtual character needs to achieve at the second moment. The device further includes: The determining module is used to determine a first difference between the first three-dimensional coordinates corresponding to the first position and the second three-dimensional coordinates corresponding to the second position; and to determine a second difference between the first yaw angle corresponding to the first orientation and the second yaw angle corresponding to the second orientation. The stitching module is further configured to stitch together the first difference, the second difference, the first roll angle and the first pitch angle corresponding to the first orientation, the second roll angle and the second pitch angle corresponding to the second orientation, the first body posture, and the second body posture to obtain a stitching vector.

19. The apparatus according to claim 17, characterized in that, The motion control model includes a rule model, which includes motion control rules. The motion control rules are characterized by a positive dynamic equation, which includes the following factors: inertia matrix, second-order time derivative of the first state vector, inertial force vector, gravity vector, and command matrix, wherein the command matrix includes internal torque command and external torque command. The motion prediction module is further configured to: determine the inertia matrix and the gravity vector based on the first state vector; determine the inertial force vector based on the first state vector and the first-order time derivative of the first state vector; determine the second-order time derivative of the first state vector based on the first state vector, the second state vector, the inertial force vector, the gravity vector, the inertia matrix, and the external torque command; substitute the second-order time derivative of the first state vector into the positive dynamic equation to obtain the internal torque command, and determine the internal torque command as the second motion command.

20. The apparatus according to claim 19, characterized in that, The device further includes: The determining module is used to determine the time difference between the second time point and the first time point, and to determine the first multiplication result of the first time derivative of the first state vector and the time difference; to subtract the second state vector from the first state vector to obtain a first subtraction result, and to subtract the first subtraction result from the first multiplication result to obtain a second subtraction result; to multiply the second subtraction result with a first parameter to obtain a second multiplication result, and to subtract the second multiplication result from the inertial force vector to obtain a third subtraction result; and to determine the... The third subtraction result is subtracted from the gravity vector to obtain a fourth subtraction result, and the fourth subtraction result is added to the external torque command to obtain a first addition result; the second parameter is multiplied by the time difference to obtain a third multiplication result, and the third multiplication result is added to the inertia matrix to obtain a second addition result; the inverse of the second addition result is determined, and the inverse is multiplied by the first addition result to obtain a fourth multiplication result; the fourth multiplication result is determined as the second time derivative of the first state vector.

21. The apparatus according to any one of claims 14 to 20, characterized in that, The device further includes: The determining module is configured to determine the selection probability by means of the following method before selecting a target action instruction for controlling the virtual character from the first action instruction and the second action instruction based on the selection probability: using the number of samples stored in the sample queue as the independent variable x, constructing the input variable of a function, wherein the range of the dependent variable y of the function satisfies 0 < <1, and y increases monotonically as the input variable increases; substitute the number of samples into the function, and determine the output value as the selection probability.

22. The apparatus according to any one of claims 14 to 20, characterized in that, The storage module is further configured to, in response to the number of samples stored in the sample queue reaching a first quantity threshold, delete the oldest sample from the sample queue in a first-in-first-out order and store the most recently generated sample.

23. The apparatus according to any one of claims 14 to 20, characterized in that, The training module is further configured to perform the following processing whenever the number of samples stored in the sample queue reaches a second quantity threshold: randomly select multiple samples from the sample queue; train the first neural network model based on the multiple samples until the difference between the first action command output by the first neural network model and the second action command output by the action control model is less than the difference threshold.

24. The apparatus according to claim 23, characterized in that, The first neural network model is a multilayer perceptron, which includes multiple layers, and each layer includes a corresponding weight matrix and bias vector; The training module is further configured to perform the following processing for each sample: determine the difference between the second action instruction and the first action instruction included in the sample, and perform backpropagation processing based on the difference to update the weight matrix and bias vector corresponding to each layer in the multilayer perceptron.

25. The apparatus according to any one of claims 14 to 20, characterized in that, The first image frame corresponding to the first time moment and the second image frame corresponding to the second time moment are two adjacent frames in the virtual scene; or, the first image frame corresponding to the first time moment and the second image frame corresponding to the second time moment are two non-adjacent frames in the virtual scene, and the third state vector that the virtual character needs to correspond to at the third time moment corresponding to the third image frame is determined by interpolation, and the third image frame is located between the first image frame and the second image frame.

26. A role control device for a neural network model, characterized in that, The device includes: The acquisition module is used to acquire the fourth state vector of the virtual character in the virtual scene at the fourth moment, as well as the control command triggered on the virtual character; The determining module is used to determine the fifth state vector that the virtual character needs to correspond to at the fifth moment, based on the control instruction vector corresponding to the control instruction and the fourth state vector. The action prediction module is used to call the first neural network model to perform action prediction processing based on the fourth state vector and the fifth state vector to obtain the third action command; Wherein, the first neural network model is trained by the training method of the neural network model according to any one of claims 1 to 12; The control module is used to control the actions of the virtual character according to the third action command.

27. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable instructions; The processor, when executing executable instructions stored in the memory, implements the training method of the neural network model according to any one of claims 1 to 12, or the role control method of the neural network model according to claim 13.

28. A computer-readable storage medium storing executable instructions, characterized in that, When the executable instructions are executed by the processor, they implement the training method of the neural network model according to any one of claims 1 to 12, or the role control method of the neural network model according to claim 13.

29. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the training method of the neural network model according to any one of claims 1 to 12, or the role control method of the neural network model according to claim 13.

Images