Data processing methods and related products

By automatically obtaining user instructions and using the control strategy model to generate virtual character animations, the problems of low efficiency and poor quality of traditional animation production are solved, efficient and high-quality animation generation is achieved, and storage and memory requirements are reduced.

CN116468827BActive Publication Date: 2024-11-26TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210028117.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-11
Publication Date
2024-11-26
Estimated Expiration
2042-01-11

AI Technical Summary

Technical Problem

In traditional methods, virtual character animations are inefficient and poor in production, making it difficult to achieve natural and realistic visual effects.

Method used

By obtaining user instructions, the target control policy model is automatically selected from the control policy model library, and the model is used to process the virtual character state, generate action control policies and generate animations to achieve automated and intelligent generation of animations.

Benefits of technology

It improves the production efficiency and quality of character animations, reduces the dependence on manual production, especially in real-time animation generation scenes, and reduces storage and memory usage.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116468827B_ABST
    Figure CN116468827B_ABST
Patent Text Reader

Abstract

The embodiment of the present application discloses a data processing method and related products, which include: obtaining a user instruction including a target action type for a target virtual character; selecting a target control strategy model belonging to the target action type from a control strategy model library; the target control strategy model is obtained by training a target action segment belonging to the target action type; using the target control strategy model to process the state of the target virtual character in the tth control cycle, and obtaining the action control strategy of the target virtual character in the t+1th control cycle, where t is a positive integer; according to the action control strategy of the t+1th control cycle, generating the action animation of the target virtual character in the t+1th control cycle and the state of the t+1th control cycle; outputting the action animation of the target virtual character in all control cycles. Through the embodiment of the present application, action animation can be automatically and intelligently generated, effectively improving the production efficiency and quality of character animation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, a data processing device, a computer equipment, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the rapid development of Internet technology, the production of game animations is increasingly pursuing natural and realistic visual effects, especially the animation of virtual characters in games. However, generating animations for virtual characters in real time is a very challenging task. The traditional method is usually for animators to manually create animations for virtual characters and store them in an animation library in memory. Manual animation is inefficient and of poor quality. Summary of the invention

[0003] The embodiments of the present application provide a data processing method and related products, which can automatically and intelligently generate action animations, effectively improving the production efficiency and quality of character animations.

[0004] On the one hand, an embodiment of the present application provides a data processing method, including:

[0005] Obtaining a user instruction for a target virtual character, the user instruction including a target action type;

[0006] A target control strategy model belonging to the target action type is selected from the control strategy model library; the target control strategy model is a control strategy model trained according to a target action segment belonging to the target action type;

[0007] Using the target control strategy model, the state of the target virtual character in the t-th control cycle is processed to obtain the action control strategy of the target virtual character in the t+1-th control cycle. The action control strategy is used to characterize the action performed by the target virtual character, and t is an integer greater than or equal to 1.

[0008] Generate the action animation of the target virtual character in the t+1th control cycle and the state of the target virtual character in the t+1th control cycle according to the action control strategy in the t+1th control cycle;

[0009] Output the target virtual character's action animation in all control cycles.

[0010] On the one hand, an embodiment of the present application provides a data processing device, including:

[0011] An acquisition module, used to acquire a user instruction for a target virtual character, wherein the user instruction includes a target action type;

[0012] A selection module is used to select a target control strategy model belonging to a target action type from a control strategy model library; the target control strategy model is a control strategy model trained based on a target action segment belonging to the target action type;

[0013] A processing module, used to process the state of the target virtual character in the t-th control cycle using the target control strategy model, and obtain the action control strategy of the target virtual character in the t+1-th control cycle, where the action control strategy is used to characterize the action performed by the target virtual character, and t is an integer greater than or equal to 1;

[0014] The processing module is further used to generate the action animation of the target virtual character in the t+1th control cycle and the state of the target virtual character in the t+1th control cycle according to the action control strategy of the t+1th control cycle;

[0015] The output module is used to output the action animation of the target virtual character in all control cycles.

[0016] On the one hand, an embodiment of the present application provides a computer device, including: a processor, a memory, and a network interface; the processor is connected to the memory and the network interface, wherein the network interface is used to provide network communication functions, the memory is used to store program codes, and the processor is used to call program codes to execute the data processing method in the embodiment of the present application.

[0017] Accordingly, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the data processing method in the embodiment of the present application is executed.

[0018] Accordingly, an embodiment of the present application provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, a data processing method provided in one aspect of an embodiment of the present application is implemented.

[0019] In the embodiment of the present application, by obtaining the user instruction, the corresponding target control strategy model can be automatically selected from the control strategy model library according to the target action type included in the user instruction, and the action control strategy is determined based on the target control strategy model, thereby generating the action animation of the target virtual character in all control cycles. The above scheme intelligently matches the target control strategy model according to the target action type. Since the target control strategy model is obtained by training the action fragment of the target action type, the action characteristics of the specific action type are accurately learned and captured, so the target control strategy model can accurately generate the action animation of the target action type to ensure the quality of the animation; the target control strategy model automatically processes the state of the virtual character, generates an animation that meets the user's expectations, abandons the manual production method, and effectively improves the efficiency of animation production; in addition, when this scheme is applied to the scene of real-time animation generation, since there is no need to store animation materials of different action types, only the trained control strategy model needs to be called to generate the animation in real time, especially in the generation of game character animation, which can effectively reduce the storage occupied by the game and the memory at runtime. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 is an architecture diagram of a data processing system provided by an exemplary embodiment of the present application;

[0022] Figure 2 It is a flowchart of a data processing method provided by an exemplary embodiment of the present application;

[0023] Figure 3 An exemplary embodiment of the present application provides a relationship between a character coordinate system and a center-of-gravity joint coordinate system of a biped character model;

[0024] Figure 4 is a schematic diagram of a target control strategy model processing data provided by an exemplary embodiment of the present application;

[0025] Figure 5 It is a flowchart of a target control strategy model training method provided by an exemplary embodiment of the present application;

[0026] Figure 6 It is a structural schematic diagram of a target control strategy model training framework provided by an exemplary embodiment of the present application;

[0027] Figure 7 It is a structural schematic diagram of another control strategy model training framework provided by an exemplary embodiment of the present application;

[0028] Figure 8 is a structural schematic diagram of a data processing device provided by an exemplary embodiment of the present application;

[0029] Fig. 9 It is a structural diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] In order to better understand the solutions of the embodiments of the present application, the relevant terms and concepts that may be involved in the embodiments of the present application are first introduced below.

[0032] 1. UE (Unreal Engine)

[0033] UE is a real-time 3D creation tool that can produce photo-realistic visual effects and immersive experiences. It can be applied in games, architecture, film and television, simulation and other industries. The dynamic simulation function provided by the UE4 (Unreal Engine 4) editor can make the generated action animation more realistic, because it will strictly simulate gravity, external force collision and the virtual character's own movement driven by muscle joints according to the laws of physics, and its control strategy is very complex: it is difficult to deduce what kind of commands should be given to the simulator from the position and posture that players or game developers expect the game character to reach, and it is difficult to obtain a control strategy based on specifications. However, there are already many examples of using deep reinforcement learning to train control strategies, such as DReCon and deep mimic (an action generation algorithm). Reinforcement learning methods can be used to train control strategies for bipedal character skeletons.

[0034] 2. Quaternions

[0035] Quaternions are simple hypercomplex numbers. Complex numbers are composed of real numbers plus the imaginary unit i, where i = -1. Similarly, quaternions are composed of real numbers plus three imaginary units i, j, and k. The geometric meaning of i, j, and k themselves can be understood as a kind of rotation, where i rotation represents the rotation of the positive direction of the X axis to the positive direction of the Y axis in the plane where the X axis and the Y axis intersect, j rotation represents the rotation of the positive direction of the Z axis to the positive direction of the X axis in the plane where the Z axis and the X axis intersect, and k rotation represents the rotation of the positive direction of the Y axis to the positive direction of the Z axis in the plane where the Y axis and the Z axis intersect, and -i, -j, and -k represent the reverse rotations of i, j, and k rotations respectively.

[0036] 3. Kinematics

[0037] It is specifically used to describe the motion of an object, that is, the change in the position of an object in space over time, without considering factors such as force or mass that affect the motion.

[0038] 4. Dynamics

[0039] It mainly studies the relationship between the forces acting on an object and the movement of the object.

[0040] 5. Deep reinforcement learning

[0041] Deep Reinforcement Learning, or DRL for short. As the name suggests, it is a combination of deep learning and reinforcement learning. In the field of gaming, deep reinforcement learning allows agents to explore the environment to learn strategies without the need for labeled samples. Among them, reinforcement learning is a kind of learning that maps environmental states to action spaces, based on the Markov Decision Process (MDP); deep learning can discover optimal strategies through reward and punishment mechanisms, focusing on learning strategies to solve problems. Deep learning uses multi-layer networks and nonlinear transformations to learn the distributed feature representation of data, focusing on the perception and expression of things.

[0042] 6. Intelligent Agents

[0043] An entity that can perceive the environment through sensors and act on the environment through actuators. A deep reinforcement learning agent is a computational entity based on a deep reinforcement learning algorithm.

[0044] 7. Artificial Intelligence

[0045] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level technology and software-level technology. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0046] The solution provided in the embodiments of the present application relates to computer vision technology and machine learning / deep learning in the field of artificial intelligence.

[0047] Among them, computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify, track and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0048] Machine Learning (ML) is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching. In the embodiments of the present application, the training of the control strategy model can utilize the technology of reinforcement learning, specifically deep reinforcement learning, such as the deepmimic algorithm (an action generation algorithm) for training and learning, and the target control strategy model finally learned can be used to generate action animations.

[0049] It will be understood that in the specific implementation of the present application, data related to the status of virtual characters, association status, other data related to virtual characters, and user instructions, when the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0050] The architecture of the data processing system provided in the embodiments of the present application will be introduced below in conjunction with the accompanying drawings.

[0051] See also Figure 1 , Figure 1 FIG. 1 is a schematic diagram of a data processing system provided by an exemplary embodiment of the present application. Figure 1 As shown, the system includes a database 100 and a data processing device 101. The data processing device 101 and the database 100 can be connected to each other by wired or wireless communication. The database 100 can be a local database of the data processing device 101 or a cloud database accessible by the data processing device 101. The data processing device 101 can be a terminal device. In one embodiment, the terminal device can be a smart phone, a tablet computer, a smart wearable device, a smart voice interaction device, a smart home appliance, a personal computer, a vehicle-mounted terminal, etc., which are not limited here.

[0052] In one embodiment, the database 100 includes a control strategy model library, which is used to store control strategy models belonging to different action types. Each control strategy model is trained using a target action segment of a certain action type, and can generate a certain action animation of a virtual character. The action animations of different action types can be walking, running, jumping, backflips, side flips, roundhouse kicks, etc. These action animations are animations that meet kinematic characteristics and dynamic characteristics, and are applied in virtual characters to have very realistic visual effects. In addition, the database 100 also includes an animation material library, which is used to store the action animations generated by the control strategy model for the virtual character. The action types included in the animation material library can be equal to or more than the action types to which the control strategy model belongs. Optionally, the animation material library can also store the action animations of virtual objects manually made by animators.

[0053] The data processing device 101 can obtain user instructions for the target virtual character. According to the target action type included in the user instruction, a target control strategy model belonging to the target action type can be selected from the control strategy model library, and then the target control strategy model is used to process the state of the target virtual character in the tth control cycle to obtain the action control strategy of the t+1th control cycle, wherein the state is used to characterize the characteristic information of the target virtual character, for example, it can be the feature extracted by encoding the original information of the current game screen through technologies such as CNN.

[0054] According to the action control strategy, the action animation and state of the target virtual character in the t+1th control cycle are generated. The target control strategy model then processes the state of the target virtual character in the t+1th control cycle to generate the action animation and state of the next control cycle. This process is repeated in this way, and the action animation of the target virtual character in all control cycles can be generated and output. Optionally, the data processing device 101 is a terminal device. In one embodiment, the terminal device can combine the action animations of all control cycles into an animation sequence in the order of generation time, and store them in an animation material library. Subsequently, the terminal device can respond to the user's animation playback instruction to extract the action animation of the target action type from the animation material library and play it in the terminal device. In another embodiment, the terminal device can also directly play the action animation of the target virtual character in the image interface after rendering and skinning when detecting the generation of the action animation of all control cycles.

[0055] The data processing system can be applied to scenarios involving the generation of action animations of virtual characters, such as in the field of games, film and television, etc. Through the target action type included in the user instruction, a target control strategy model of the target action type can be selected from the control strategy model library, and an action animation corresponding to the target action type can be generated according to the target control strategy model, so that the action animation can be generated efficiently, and the quality of the generated action animation can be effectively improved by using the trained target control strategy model.

[0056] The specific implementation of the data processing method proposed in the embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0057] See also Figure 2 , Figure 2 is a flowchart of a data processing method provided by an exemplary embodiment of the present application. The method can be performed by a computer device (e.g. Figure 1 The data processing device 101 shown in the figure is executed. The data processing method includes but is not limited to the following steps.

[0058] First, the scenarios to which the data processing solution provided in the embodiment of the present application is applicable are briefly introduced. In one embodiment, this data processing solution can be applied to the following two animation generation scenarios, namely:

[0059] 1) Animation pre-generation scenario: This scenario is suitable for users who design action animations, such as animators, artists, etc. The user who designs action animations initiates user instructions by setting corresponding parameters in the client (such as the frame rate of the action animation, the target action type of the action animation, etc.). Through the pre-set target action type, the corresponding target control strategy model is called from the control strategy model library to generate an action animation of the target action type. The action animation can be stored in the animation material library so that when an animation playback instruction initiated by a user of the action animation of this action type is received later, the action animation of the target action type is called from the animation material library for display.

[0060] 2) Real-time animation generation scenario: This scenario is suitable for users who use action animation, such as game players. When receiving a user instruction initiated by such a user, according to the target action type included in the user instruction, the target control strategy model belonging to the target action type can be called from the control strategy model, and the action animation of the target virtual character can be generated in real time according to the target control strategy model, and the action animation is rendered and skinned, and displayed in the user interface. For more detailed implementation methods, please refer to the following content.

[0061] S201, obtaining a user instruction for a target virtual character.

[0062] The target virtual character refers to the virtual character for which the action animation is to be generated. The target virtual character may be a virtual character in games, cartoons, movies, television, etc. For example, the target virtual character is a game character. Virtual characters include but are not limited to humanoid characters, simulated robots, animal characters (such as dinosaurs, lions), etc. These virtual characters may be obtained by simulating and drawing people or objects in a real environment.

[0063] Based on the above introduction to the animation generation scenario, when this solution is applied to the animation pre-generation scenario, the user instructions for the target virtual character obtained can be generated by the user who designs the action animation (such as an artist) performing an operation trigger in the client; when this solution is applied to the animation real-time generation scenario, the user instructions for the target virtual character obtained can be generated by the user who uses the action animation (such as a player) performing an operation trigger in the client. The client can run in a terminal device. For example, the user triggers it by clicking the physical keyboard or mouse of the terminal device, or by screen touch, such as gesture sensing (such as long press, single click, continuous click), or voice sensing, body sensing, etc. The triggering method of the user instruction is not limited here.

[0064] The acquired user instruction includes a target action type. The target action type can be set directly by the user (mainly used for pre-generated action animation scenes) or indirectly selected through a mapping identifier (mainly used for scenes using action animations), so that the target action type can be the action type expected by the user, and can be used to instruct the target virtual character to ultimately generate an action animation that meets the user's expectations. Action types can include running, walking, jumping, backflips, roundhouse kicks, etc., and each action type corresponds to a piece of action animation. In the process of generating action animations, the client can be instructed to select which control strategy model from the control strategy model library as the target control strategy model according to the target action type, that is, the following step S202.

[0065] S202, selecting a target control strategy model belonging to a target action type from a control strategy model library.

[0066] The control strategy model library is used to store trained control strategy models of different action types. Each control strategy model is trained using action clips of a certain action type. The action clips can be collected by motion capture, or designed by artists, or generated by various intelligent action generation algorithms, or obtained from the Internet. There is no restriction on the source of the action clips. It should be noted that the action clip may not conform to the characteristics of dynamic simulation and may not look realistic enough, but it may also conform to the characteristics of dynamic simulation, such as action clips directly derived from real videos, and there is no restriction on this.

[0067] It can be seen that there is a mapping relationship between the control strategy models and action types stored in the control strategy model library. Therefore, according to the target action type included in the user instructions for the target virtual character, a target control strategy model belonging to the target action type can be selected from the multiple control strategy models included in the control strategy model library. The target control strategy model is a control strategy model trained according to the target action segment belonging to the target action type. The target action type here is the target action type in the user instruction, which is an action type that matches the input target action type among all the action types included in the control strategy model library, such as a backflip. By selecting a target control strategy for generating an action animation that meets the user's expectations from the control strategy model library, the action animation can be generated in real time or in advance according to the following steps. For the specific training process of the target control strategy model, please refer to Figure 5 The description of the corresponding embodiment will not be described in detail here. It should be noted that all control strategy models stored in the corresponding control strategy model can be used as follows Figure 5 The training method of the target control strategy model is implemented.

[0068] Here, the trained control strategy model is matched with the action type. Although multiple control strategy models are trained through action clips of different action types in the training stage, rather than generalizing a control strategy model for all action types, in the control strategy model application stage, there is no need to input the target action clip for processing. Instead, the control strategy model and the action type are directly matched, and the action animation is obtained by using the target control strategy of the target action type, thereby effectively reducing the data processing amount of the target control strategy model.

[0069] In one embodiment, the control strategy model may include a strategy neural network (or strategy network), which may be a fully connected neural network (Multi-Layer Perceptron, MLP) based on a deep reinforcement learning algorithm (such as a deep mimic algorithm or), or a convolutional neural network, or a residual neural network, etc., and the type of the strategy neural network is not limited here. For content on training a control strategy model according to a target action segment of a target action type, please refer to Figure 5 The description in the corresponding embodiment will not be described in detail here. Optionally, the control strategy model can be set in the client's motion simulation engine (such as UE4) program, and the physical simulator of the motion simulation engine can be used for reinforcement learning training, which helps to generate complex control strategies, so as to use the trained target control strategy model to generate high-quality animations with dynamic simulation, and by setting the control strategy model in the motion simulation engine for training, the target strategy model processing data does not require additional programming, and can be automatically compatible with the motion simulation engine program, which is more convenient for developers to use and maintain.

[0070] S203, using the target control strategy model, processing the state of the target virtual character in the t-th control cycle to obtain the action control strategy of the target virtual character in the t+1-th control cycle, where t is an integer greater than or equal to 1.

[0071] The t-th control cycle belongs to the current control cycle of the target virtual character. The control cycle can characterize the time of controlling the simulated action of the target virtual character, such as 0.1s. The time of each control cycle can be the same or different. For example, the first control cycle is 0.1s, the second control cycle may be 0.2s, and the third cycle may be 0.05s. This may depend on the amount of data processed by the control strategy model. The control cycle can also be independent of time, and only means that a set of events repeat in the same order. Repeating once is called a control cycle. The target virtual character has a corresponding state in each control cycle. The state can characterize the characteristic information of the target virtual character, and the state of each control cycle can be obtained by the client. Optionally, the state can include a variety of data describing the target virtual character information, such as joint position, joint posture and joint angle. It should be noted here that the state of the target virtual character obtained by the client in each control cycle is not the most original data of the target virtual character, but is calculated and sorted from the original data obtained.

[0072] In one embodiment, the client can collect the initial state information of the target virtual character in the tth cycle. The initial state information is the most original data of the target virtual character in the current control cycle. Then, the state in the current control cycle can be determined based on the initial state information, and the state can be organized into a data format that conforms to the target control strategy model processing. For example, the state of the target virtual character in the tth control cycle is a multidimensional vector organized in a preset format.

[0073] Optionally, the initial state information may include information such as the current character state, environmental parameters, and the current simulation time, wherein the current character state is used to reflect the state of the target virtual character in the current processing cycle, and the current character state may include the position information of the target virtual character (hereinafter referred to as the character for the convenience of description), for example, it may include the position and posture of the character in the world coordinate system, the position and posture of the character's center of gravity joint (such as pelvis joint) relative to the character, and the position of all skeletal joints participating in the physics simulation relative to the character, and the above positions or postures can be represented by quaternions; environmental parameters are used to characterize the virtual environment in which the target virtual character is located, and may be environmental parameters related to deep reinforcement learning; the current simulation time refers to the time used for integration in the dynamic simulation, rather than the time of the world clock.

[0074] For example, Figure 3The relationship between the character coordinate system and the center of gravity joint coordinate system of a biped character model in a dynamics simulation engine (such as UE4) is shown. The character coordinate system is the world coordinate system, which can describe the position of the target virtual character in the world coordinate system. The center of gravity joint coordinate system is used to describe the movement position of the center of gravity joint of the biped character model. Operations in the center of gravity joint coordinate system can drive the center of gravity joint to move to achieve the desired position. The center of gravity joint here can be the pelvis joint.

[0075] In the above data, the posture of all skeletal joints participating in the physics simulation (i.e., dynamics simulation) relative to the character can obtain the rotation of all skeletal joints participating in the dynamics simulation relative to their parent joints, i.e., the joint angles. By differencing the joint angles, all state information including speed and angular velocity can also be inferred.

[0076] Taking the bipedal human skeleton model as an example, in the bipedal human skeleton model, in addition to the pelvis joint, there are 18 rotatable skeletal joints, each of which has degrees of freedom of different dimensions, as shown in Table 1 below.

[0077] Table 1 Rotatable joints and degrees of freedom

[0078]

[0079] It can be found that the degrees of freedom of each skeletal joint range from 1 to 3 dimensions, and the 18 skeletal joints can generate a total of 32 effective degrees of freedom by converting the rotation angle. Therefore, after the client obtains the initial state information, it can use the 18 groups of quaternions representing the rotation of the skeletal joints in the current character state to calculate the 32 angle values ​​representing the rotation of the skeletal joints, that is, the joint angles.

[0080] The current state of the target virtual character (hereinafter referred to as the character state) can be fully characterized based on the joint angle, joint position and joint posture. For example, the joint position is the center of gravity joint position, the joint posture is the center of gravity joint posture, and the center of gravity joint position is represented by the 3D position coordinates of the center of gravity joint pelvis in the world coordinate system, and the center of gravity joint posture is the posture of the center of gravity joint relative to the character, represented by a 6-dimensional forward-up vector (for example Figure 3 Thus, 32-dimensional joint rotations, 3-dimensional center-of-gravity joint positions, and 6-dimensional center-of-gravity joint postures, a total of 41-dimensional vectors, can be included in the state of the target virtual character.

[0081] Optionally, the state of the target virtual character in the t-th control cycle includes the character states of multiple control cycles, and the character state can be represented by joint angles, joint positions, and joint postures, and is a multi-dimensional vector. Exemplarily, the character state of each control cycle is a 41-dimensional vector, and the state of the target virtual character in the t-th control cycle is set to a 41×7=287-dimensional vector, which includes the character state of the target virtual character in the t-th control cycle, and the character state of the target virtual character in historical control cycles, such as the t-1th control cycle and the t-2th control cycle, and the values ​​of the remaining 4 dimensions can be set to 0 or other random constants, that is, the character states of these 7 control cycles are combined together to form the state of the target virtual character in the t-th control cycle.

[0082] In another embodiment, the initial state information may also include an action segment of the target action type, and the relevant parameter information of the action segment of the target action type is similar to the content included in the character state of the target virtual character in each control cycle. For example, the current state of the target virtual character includes a vector of 41 dimensions, including joint angles, center of gravity joint positions, and center of gravity joint postures. Then, the relevant parameter information of the target action segment may also include a vector of 41 dimensions, including joint angles, center of gravity joint positions, and center of gravity joint postures. All of this information can be used to describe the target posture of the reference virtual character in the target action segment, and the target posture is the posture tracked by the target virtual character in the next one or more control cycles. In order to ensure the best processing effect of the target control strategy model, the relevant parameter information of the target action segment and the character state of the target virtual character can be arranged together as an observation vector, that is, the state of the t-th control cycle, and then processed by the target control strategy model. Exemplarily, taking the tth control cycle as a reference, the character states of the t+1th control cycle, t+2th control cycle, t+5th control cycle and t+10th control cycle are selected in the target action segment, and the character states of the target virtual character in the t-1th control cycle and t-2th control cycle are selected, and the character state of the target virtual character in the t control cycle is combined as the observation vector, wherein the character state of each control cycle includes 32-dimensional joint angles, 3-dimensional joint positions and 6-dimensional joint postures, constituting 41-dimensional data, and the character states of 7 different control cycles constitute a 287-dimensional observation vector, which is used to represent the state of the target virtual character in the tth control cycle, which includes values ​​that are not set to 0 or other constants.

[0083] That is to say, the state of the t-th control cycle may include joint angles, joint positions, and joint postures. Other parameters (such as environmental parameters, current simulation time) may also be included in the state, but subsequent processing mainly focuses on the data of joint angles, joint positions, and joint postures. Optionally, these data can be sorted and arranged in the format of observation vectors, and then input into the target control strategy model for processing, and the action control strategy of the target virtual character in the t+1-th control cycle is output. The specific processing method can be found in the following introduction.

[0084] See also Figure 4 , is a schematic diagram of a target control strategy model processing data provided in an embodiment of the present application. The client uploads the state of the target virtual character to the target control strategy model, which then calculates and obtains the control command, namely the action control strategy.

[0085] The action control strategy is used to characterize the action performed by the target virtual character. The action control strategy is also called the control command action. Specifically, in the dynamics simulation engine, the target virtual character with dynamics simulation can be guided to complete the predetermined action indicated by the control command. Taking the target virtual character as a bipedal human skeleton model as an example, the action represented by the action control strategy can be represented by the angle value that each skeletal joint in the target virtual character should rotate in the next cycle, that is, the joint angle of the t+1th control cycle. At the same time, the joint angle can also be used as the data included in the state of the t+1th control cycle.

[0086] S204: Generate, according to the action control strategy of the t+1th control cycle, an action animation of the target virtual character in the t+1th control cycle and a state of the target virtual character in the t+1th control cycle.

[0087] The action control strategy of the t+1th control cycle is executed after the end of the tth control cycle, and the data generated by the target virtual character performing the action acts on the t+1th control cycle. That is, after the end of the tth control cycle, the target virtual character moves in the t+1th control cycle according to the action control strategy generated by the state of the tth control cycle, and the corresponding action animation can be generated. At this point, the target virtual character completes the action according to the action control strategy in the t+1th control cycle, and the information such as the posture and position of the target virtual character has changed, that is, the state of the target virtual character has changed, so that the target virtual character has a new state, that is, the state of the t+1th control cycle.

[0088] In one embodiment, the specific implementation of step S204 may be as follows:

[0089] 1) Determine the rotational torque of each target rotating object in the target virtual character according to the state of the t-th control cycle and the action control strategy of the t+1-th control cycle.

[0090] The target virtual character includes multiple target rotation objects. Taking the target virtual character as a bipedal character model as an example, the multiple target rotation objects included in the target virtual character are multiple rotatable joints, such as the 18 rotatable joints shown in Table 1 above. In order to make the target virtual character conform to the dynamic characteristics, the motion control strategy can be applied to the dynamic simulation calculation of the physical simulator (i.e., the dynamic simulator) in the motion simulation engine (such as UE4) until the end of the tth control cycle. The rotational torque of each target rotation object is also called torque, which can be specifically expressed as follows (1):

[0091] Formula (1)

[0092] Among them, spring and damping are two control system coefficients, which are generally selected as constants 10000 and 2000 by default. Users can adjust the values ​​to achieve a more ideal control effect. x is the angular velocity of the joint angle in the current control cycle, q x Represents the current joint angle, where x is the continuous time. The angular velocity of the joint angle can be obtained by differentiating different joint angles. x and q x Belongs to the state within the tth control cycle.

[0093] 2) According to the rotational torque, each target rotation object of the target virtual character is controlled to move in the t+1th control cycle to generate the action animation of the target virtual character in the i+1th training cycle.

[0094] According to the calculated torque, each rotatable joint can rotate according to the corresponding torque, and the rotation of each rotatable joint constitutes the movement of the target virtual character in the t+1th control cycle, thereby generating an action animation. Therefore, after the target virtual character's movement ends, a new state of the target virtual character is generated and used as the state of the target virtual character in the t+1th control cycle.

[0095] 3) After the target virtual character finishes moving, the state of the target virtual character in the t+1th control cycle is obtained.

[0096] In one embodiment, the method of obtaining the state of the target virtual character in the t+1th control cycle includes: obtaining the environment information and posture information of the target virtual character; generating the state of the target virtual character in the t+1th control cycle according to the posture information and the environment information. Among them, the environment information is used to characterize the virtual environment in which the target virtual character is located; the posture information is used to characterize the position and posture of the target virtual character. The target virtual character moves according to the action animation generated by the action control strategy, for example, the action animation of taking a step with the right foot when walking, and the movement ends, that is, the corresponding posture indicated by the action control strategy is reached. At this time, the position and posture of each rotatable target object of the target virtual character, such as each rotatable joint, have changed, that is, the posture information obtained here is new posture information compared with the previous cycle, and the environment information and posture information correspond to the above initial state information, for example, the posture information includes the position and posture quaternion of the character (component) in the world coordinate system, the position and posture quaternion of the character pelvis joint relative to the component, and the posture quaternion of all skeletal joints participating in the physics simulation relative to the component. For determining the state according to the initial state information collected in the t+1th control cycle, reference may be made to the introduction in step S203 , which will not be elaborated here.

[0097] It should be noted that the processing of the state in the t+1th control cycle is also implemented in accordance with the aforementioned processing of the state in the tth control cycle, and the process is repeated in sequence until the state of the preset number of control cycles is processed. After the result is obtained, the action animation of all control cycles can be obtained. When applied in a pre-generated animation scene, the preset number can be determined by the user who designs the action animation, such as an animator or game artist; when applied in a real-time generation scene, the preset number can also be included in the user instruction, that is, it is determined by the game scene in which the target virtual character is located and the player's operation.

[0098] By applying the motion control strategy to the physical simulator of the motion simulation engine to perform dynamic simulation calculations, the rotational torque of each rotatable target object is obtained, and then movement is performed according to the rotational torque, which can generate high-quality motion animations with more realistic animation effects and satisfying dynamic characteristics.

[0099] S205: Output the action animation of the target virtual character in all control cycles.

[0100] After the state of the next control cycle is obtained according to the state of the target virtual character in the current control cycle, step S204 is repeated to generate action animations for all control cycles. The action animation of each control cycle is a frame of animation data, and the action animations of all control cycles can constitute an action segment (or character animation) with a certain duration, such as a 3s walking segment, a 2s backflip. It should be noted here that the action animations of all cycles generated according to the action control strategy output by the target control strategy model belong to the target action type, that is, the target action type included in the user instruction, so that the action animation generated meets the user's expectations. Afterwards, the action animations of the target virtual character in all control cycles can be output, and more details can be found in the following introduction.

[0101] In this way, through the above steps, the target control strategy model automatically generates an animation sequence of the target action type. If the user who designs the action animation repeats the above process and inputs different action types, action animations of different action types can be efficiently obtained, thereby improving the production efficiency of action animations of different action types, and greatly reducing the workload of animators in manually producing action animations of target virtual characters. When applied in the game field, the efficient generation method of action animations of different action types can further reduce game development work and shorten the development process.

[0102] In one embodiment, an optional implementation of step S205 may be:

[0103] The action animations of all control cycles are combined into an animation sequence in the order of recorded generation time from small to large, and the animation sequence is stored in the animation material library; the action type of the animation sequence belongs to the target action type; when an animation playback instruction containing the target action type is received, the animation sequence is retrieved from the animation material library in response to the animation playback instruction for playback.

[0104] The action animation of each control cycle is recorded with a generation time, which can be a time point under the world clock. According to the recorded generation time, the action animations of all control cycles can be sorted to form an action sequence with a chronological order, which can be an action in a continuous action space. Afterwards, the animation sequence can be stored in the animation material library and marked as a target action type, so that when an animation playback instruction containing the target action type is received, the animation sequence of the target action type is retrieved from the animation material library in response to the animation playback instruction for playback.

[0105] The animation sequences processed by different target control strategy models can be stored in the animation material library. Each animation sequence corresponds to an action type. Optionally, all animation sequences in the animation material library are animation sequences generated by different target control strategy models. When the user initiates an animation playback instruction, the user can retrieve the animation sequence of the corresponding action type from the animation material library for playback according to the action type included in the animation playback instruction.

[0106] In this way, the action sequence corresponding to the target virtual character is automatically generated by the target control strategy model and stored in the animation material library, and the action sequence of the corresponding action type is matched when the user initiates the animation playback command. This makes it convenient for users who need to use action animation to quickly and accurately find the action sequence of the target action type from the animation material library for playback.

[0107] In another embodiment, an optional implementation of step S205 may also be:

[0108] The action animations of the target virtual character in all control cycles are rendered separately according to the set rendering frame rate, and the skin mesh is applied to the rendered action animation to obtain the action animation to be played; the action animation to be played includes multiple animation frames; the multiple animation frames are played in the graphical interface, and each of the multiple animation frames is displayed after the synchronization waiting time is reached; the synchronization waiting time is determined according to the set rendering frame rate.

[0109] In one embodiment, after the action animation of all control cycles of the target virtual character is generated, the action animation can be rendered and skinned and then played directly in the graphical interface. In another embodiment, it can also be that after each action animation of a control cycle is obtained, the action animation of the control cycle is rendered and skinned, and displayed in the graphical interface after reaching the synchronization waiting time. Without waiting for the animation play instruction initiated by the user, it is suitable for generating action animation scenes in real time. Wherein, the set rendering frame rate refers to the frame rate of the action sequence to be rendered, and the higher the number of frames, the more action animation content the action sequence contains. By rendering the action animation of all control cycles of the target virtual character respectively, the action animation after rendering is obtained, and then the action animation after rendering is skinned, so-called skinning is the process of connecting the three-dimensional skeleton vertex to the position of the skeleton. Specifically, the skinned mesh can be applied to the action animation after rendering, and the skinned mesh refers to the triangular mesh wrapped around the skeleton, and one or more skeletons of the target virtual character can control each vertex of the mesh. The action animation to be played so constituted is a skeleton skinned animation, and the skeleton controls the skinning movement, and the movement of the skeleton itself is realized by the action control strategy. Afterwards, when playing multiple animation frames included in the action animation to be played on the graphical interface, there is also a synchronization waiting time. Each frame of animation is displayed on the graphical interface after the synchronization waiting time, which can make the action animation more consistent with the real time rate.

[0110] For example, taking an action animation with a frame rate of 120 frames as an example, the action animation of each control cycle in the action sequence is rendered separately, then the time required to calculate each frame of the action animation (i.e., each animation frame) may be much less than 1 / 120 second. In order to make the final played picture look consistent with the real time rate, it is forced to wait until 1 / 120 seconds before playing the animation frame. This is also called a time synchronization waiting mechanism. With this time waiting synchronization mechanism, the animation frames played in the graphical interface can be consistent with the real time rate, making it more vivid and realistic.

[0111] In one embodiment, the above-mentioned rendering, skinning and other processing are functions of the action simulation engine used for deployment, that is, the action simulation engine used for deployment has graphics rendering and a graphical interface, so that the action animation of the target virtual character can be vividly displayed to the user, and the action simulation engine (such as UE4) used to train the control strategy model and used to deploy the target control strategy model has different environments. The action simulation engine (such as UE4 program) used to train the control strategy model does not have graphics rendering and a graphical interface for displaying animations in order to speed up data collection and improve training efficiency, and does not have the above-mentioned waiting mechanism. Once the dynamic simulator in the engine calculates the latest status after each training cycle (for example, 0.1 seconds), it is immediately packaged and sent to the server. This is also to maximize the training efficiency.

[0112] The scheme provided by the embodiment of the present application, by obtaining user instructions, can automatically and intelligently select the corresponding target control strategy model from the control strategy model library according to the target action type included in the user instruction. Since the target control strategy model is obtained based on the action fragment training of an action type, the action characteristics of a specific action type are accurately learned and captured, so the target control strategy model can accurately generate the action animation of the target action type to ensure the animation quality; the target control strategy model automatically processes the state of the virtual character, generates an animation that meets the user's expectations, abandons the manual production method, and effectively improves the animation production efficiency; and in the process of the target control strategy model processing data, by calculating the torque of the target virtual character, the target virtual character action animation can be physically constrained, in accordance with the corresponding physical laws, thereby generating a more realistic action animation. In addition, when this scheme is applied to the scene of real-time animation generation, since there is no need to store animation materials of different action types, only the trained control strategy model needs to be called to generate animation in real time, especially in the generation of game character animation, the storage occupied by the game and the memory at runtime can be effectively reduced. In addition, when outputting the action animation of all cycles, there is a time synchronization waiting mechanism, which can further improve the effect of action animation playback.

[0113] In order to better understand the content of this solution, the training process of the target control strategy model is described in detail below. Figure 5 , is a flow chart of a target control strategy model training method provided in an embodiment of the present application, which is executed by a client and includes but is not limited to the following steps.

[0114] S501, obtaining the association state of the target virtual character in the i-th training cycle; wherein i is an integer greater than or equal to 1.

[0115] The training cycle is the time step of the control strategy model during the training period, for example, 0.1s. During the training phase of the control strategy model, each training cycle is the same. The i-th training cycle is the time cycle currently being trained. The associated state of the i-th training cycle may include the state of one or more previous historical training cycles. It is understandable that if it is the first training cycle, then the associated state only includes the state of the target virtual character in the first training cycle. These associated states are used to train the action control strategy of the target virtual character, and may include the data of the target virtual character itself and / or the data of the reference virtual character in the target action clip.

[0116] In one embodiment, the associated state of the i-th training cycle includes: a first state of the target virtual character, and a second state of the reference virtual character in the target action segment; the first state includes the state of the target virtual character in the i-th training cycle, and the state of the target virtual character in the historical training cycle; the second state includes the state of the reference virtual character in the reference training cycle; the reference training cycle is determined by the i-th training cycle and the set number of training cycles; the state includes position information, posture information and joint angle.

[0117] Specifically, the target action clip is a given fixed animation clip, which belongs to a target action type. The target action type here is any one of a plurality of action types, such as a roundhouse kick. The target action clip can be collected by motion capture, or can be hand-designed by an artist, or can be generated by various intelligent action generation algorithms. The target action clip may not conform to the dynamic simulation characteristics and may not look realistic enough, but may conform to the dynamic simulation characteristics, and there is no limitation on this. The task of the control strategy model training is to make the target virtual character with dynamic simulation in the action simulation engine track the target action clip as much as possible, and to use the action control strategy to control the action animation generated by the target virtual character to conform to the dynamic characteristics.

[0118] The action performed by the reference virtual character in the target action clip is the action tracked by the target virtual character. Therefore, the reference virtual character also has a state, which is specifically represented by the second state described below. The first state in the associated state is a state directly associated with the target virtual character. The first state includes not only the state of the target virtual character in the current training cycle, that is, the state of the i-th training cycle, but also the state of the historical cycle. The historical training cycle is selected based on the i-th training cycle, such as the state of the previous training cycle (i-1 training cycle) and the state of the previous two training cycles (i-2 training cycles). Of course, i here is greater than or equal to 3, and the corresponding selected historical training cycle is meaningful. Here, the state of the target virtual character in the historical training cycle is included in the first state, which can enable the subsequent control strategy model to strengthen the learning and training of historical action animations to avoid forgetting.

[0119] The second state is a state directly associated with the reference virtual character, and the state of the reference virtual character in the reference training cycle in the second state includes multiple future training cycles determined based on the i-th training cycle. Among them, the number of training cycles is set to a value such as 1, 2, 3, etc. For example, the next training cycle (i.e., the i+1th training cycle), the next two training cycles (i.e., the i+2th training cycle), etc. based on the current training cycle. Both the state of the target virtual character and the state of the virtual character in the target action clip contain position information, posture information, and joint angles. Using the state in the target action clip as a reference, the target virtual character can know the angles of rotation of each joint in the next training cycle based on the state of the current training cycle and the historical training cycle, so as to better perform tracking training.

[0120] In one embodiment, according to the above Figure 2 In the corresponding embodiment, the description of obtaining the state of the target virtual character, the specific contents included in the first state and the second state here can also be obtained in the same way. For example, the obtained first state and the second state can respectively include the following contents:

[0121] 1) First state

[0122] 41-dimensional current (i-th training cycle) state: p i , o i ,q i

[0123] 41-dimensional state of the previous cycle (i-1th training cycle): p i-1 , o i-1 ,q i-1

[0124] 41-dimensional state of the second cycle (i-2th training cycle): p i-2 , o i-2 ,q i-2

[0125] 2) Second state

[0126] The state of the next cycle (i+1th training cycle) of the 41-dimensional target action segment is: p g i+1 , o g i+1 ,q g i+1

[0127] The state of the second cycle (i+2th training cycle) under the 41-dimensional target action fragment is: p g i+2 , o g i+2 ,q gi+2

[0128] The state of the five cycles (i+5th training cycle) under the 41-dimensional target action fragment is: p g i+5 , o g i+5 ,q g i+5

[0129] The state of the ten cycles (i+10th training cycle) under the 41-dimensional target action fragment is: p g i+10 , o g i+10 ,q g i+10

[0130] Among them, p represents the joint position (such as pelvis joint position), o represents the joint posture (such as pelvis joint position), q represents the joint angle, and 41 dimensions represent 3-dimensional joint position, 6-dimensional joint posture, and 32-dimensional joint angle.

[0131] All states included in the associated states processed in the i-th training cycle in the above example are organized into a 287-dimensional vector (41×7=287), which can be called an observation vector observation. It should be noted that the first and second states are only examples, and states of other training cycles can also be selected. For example, it can be the state of the target action segment in the i+2th and i+4th training cycles.

[0132] S502, using the control strategy model to process the associated state of the i-th training cycle, obtain the action control strategy of the target virtual character in the i+1-th training cycle, and determine the action reward according to the associated state of the i-th training cycle.

[0133] In an embodiment of the present application, the control strategy model is stored in the action simulation engine of the client. The control strategy model may include a policy network trained based on a deep reinforcement learning algorithm, and the policy network may specifically be a basic neural network, such as a convolutional neural network, a residual neural network, a fully connected neural network, and the like. The action simulation engine of the client is an engine with a dynamic simulation function, such as an engine with a dynamic simulation of UE4, and the action simulation engine can be started on the CPU core of the client.

[0134] After the client collects the associated data of the target virtual character in the i-th training cycle, the control strategy model processes the associated state of the i-th training cycle and obtains the action control strategy applied to the i+1-th training cycle. At the same time, the action reward can also be determined using the associated state of the i-th training cycle. The action reward can be used to evaluate the application of the action control strategy of the i-th training cycle to the target virtual character. The learning goal can be continuously adjusted through the action reward, thereby generating an action control strategy that meets expectations.

[0135] In one embodiment, the step of processing the i-th training cycle using the control strategy model may include: determining the action expectation and action variance of the target virtual character based on the associated state of the target virtual character in the i-th training cycle and the control strategy model; determining the action control strategy distribution of the target virtual character in the i+1-th training cycle based on the action variance and the action expectation; and sampling the action control strategy distribution in the i+1-th training cycle to obtain the action control strategy of the target virtual character in the i+1-th training cycle.

[0136] Here, the control strategy model includes a simple three-layer perceptron neural network as an example, and the principle of control strategy model processing is explained as follows in combination with the following expression. Among them, the three-layer perceptron neural network is a multi-layer perceptron (Multi-Layer Perceptron, MLP), including an input layer, a hidden layer, and an output layer. The expression processed by the neural network can be recorded as:

[0137] Formula (2)

[0138] Among them, W0, W1, W2 are the parameter matrices of each layer in the control strategy model, b0, b1, b2 are the bias vectors of each layer in the control strategy model, the parameter matrix and the bias vector are collectively referred to as the strategy weight (or the weight of the strategy neural network), observation represents the association state of the i-th training cycle. Mean is a 32-dimensional vector, which is the action expectation output by the neural network.

[0139] Then the mean vector can be sampled through Gaussian distribution to obtain the action control strategy action, that is, the control command, and the control command action obeys Gaussian distribution, that is:

[0140] action ~N(mean, std) formula (3)

[0141] Among them, action is the target joint angle, which can be recorded as , mean is the action expectation, and std is the action variance. This expression represents the action control strategy distribution determined based on the action expectation and action variance. In some embodiments, the action variance can be a constant output by the control strategy model, that is, the MLP network can output mean and std at the same time. In other embodiments, the action variance can also be sent from the server to the client.

[0142] Optionally, observation is a 287-dimensional vector sorted according to the first state and the second state of the above example, the control strategy model first obtains a mean vector of 32 dimensions, and the final output action is also a 32-dimensional vector.

[0143] Afterwards, through the probability density function of the Gaussian distribution, we can get the probability of randomly sampling the action under the mean vector output by the policy network, which is called the policy sampling probability. This policy sampling probability can also be used in the training of the control policy model later.

[0144] In one embodiment, the method for determining the action reward based on the associated state of the i-th training cycle can be: differential processing is performed on the state of the target virtual character in the i-th training cycle and the state of the reference virtual character of the target action segment in the i-th training cycle to obtain differential information; the reward component corresponding to the differential information is determined; the reward component includes a position reward component, a posture reward component and a joint angle reward component; the position reward component, the posture reward component and the joint angle reward component are weightedly summed to obtain the action reward of the target virtual character.

[0145] The calculation of the action reward here introduces the target action segment. Based on the target action segment, the trajectory error of the target virtual character and the posture in the target action segment can be converted into an action reward. Specifically, referring to the state introduced in the aforementioned step S501, including position information, posture information and joint angle, the state of the target virtual character and the reference virtual character in the target action segment can be differentially calculated. For example, the position information of the target virtual character in the i-th training cycle and the position information of the target action segment in the i-th training cycle are subtracted, and the corresponding posture information and joint angle are similarly obtained to obtain various corresponding differential information. In this way, the reward component can be further determined based on the differential information, for example, the differential information is directly used as the reward component, or the differential information is processed, such as squared, exponential, etc. as a reward component. Optionally, the content included in the corresponding state of the reward component can include a position reward component, a posture reward component and a joint angle reward component, and then each reward component is assigned a corresponding weight and then summed to obtain the final action reward. Here, the weighted sum of each reward component can be used to measure the impact of different reward components on the action reward. The larger the weight, the greater the impact of the reward component on the action reward. In this way, the action reward can focus on more important reward components, thereby better learning strategies.

[0146] For example, according to the introduction of the action reward principle and the introduction of the associated state, the expression of the action reward is designed as follows:

[0147] Formula (4)

[0148] in, , , Represents differential information; , , Represent the joint angle reward component, position reward component, and posture reward component respectively; , , is the reward weight. The smaller the difference information is, the higher the action overlap between the target virtual character and the target action segment is, and the greater the action reward is. For example, the reward weights can be set to 0.7, 0.2, and 0.1, respectively, that is, the joint angle reward component determines the action reward more.

[0149] Referring to the content of the above example, the input and output of the control strategy model deployed in the action simulation engine on the client, and the specific reward calculation can be summarized as shown in Table 2 below.

[0150] Table 2 Input and output of control strategy model

[0151]

[0152] During the training phase of the control strategy model, the client's control strategy model input, output and reward can be executed according to the design in Table 2.

[0153] It should be noted that before executing step S503, the process further includes: applying the motion control strategy to the target virtual character to determine the torque of each rotatable joint in the target virtual character. Figure 4 Corresponding to formula (1) in the embodiment, the corresponding data is replaced by the target joint angle action and the current joint angle q of the target virtual character. c and the angular velocity v c In this way, it is easy to collect the associated state of the target virtual character in the i+1th training cycle.

[0154] The above two steps are what the action simulation engine of the client completes in one training cycle (eg, within 0.1 s) using the control strategy model, and this cycle is repeated, and after each training cycle is interpreted, the following step S503 is executed.

[0155] S503: Combine the associated state of the i-th training cycle, the action control strategy of the i+1-th training cycle, the action reward, and the associated state of the i+1-th training cycle into a training sample.

[0156] The client collects the association state of the target virtual character in the i-th training cycle, the action control strategy of the i+1-th training cycle output by the control strategy model, the action reward, and the association state of the i+1-th training cycle, which can be combined into a training sample, which will be used to train the control strategy model later to update the model parameters. Among them, the association state of the i+1-th training cycle is collected after the action control strategy of the i+1-th training cycle is applied to the target virtual character.

[0157] Here, the training sample is recorded as Train={observation i , a g i+1 , r i , observation i +1}, corresponding to the association state of the current training cycle, the action control strategy of the i+1th training cycle, the action reward, and the association state of the i+1th training cycle. These data are organized into a set of training samples and then sent to the server. The data uploaded to the server can vary slightly depending on the type of deep reinforcement learning algorithm selected, but these are the main data. Optionally, the control policy model training can use the deep mimic algorithm, in which the proximal policy optimization (PPO) algorithm can be used for policy optimization.

[0158] Optionally, the training samples also include a strategy sampling probability, which is generated after sampling the action control strategy distribution. For example, the value obtained after sampling the action control strategy that obeys the Gaussian distribution is the target angle passed to the action simulation engine for physical simulation. The target angle is substituted into the Gaussian distribution probability density function to obtain the sampling probability of the action. Based on the strategy sampling probability, the loss of the control strategy model can be calculated to better evaluate and adjust the control strategy model.

[0159] Optionally, the training sample also includes the current simulation time, which is the time used for integration in the dynamics simulation. The current simulation time can be used to record the order in which the data included in the state are generated.

[0160] S504: Send the training sample to the server, so that the server trains the control strategy model stored in the server according to the training sample.

[0161] The client sends the training samples to the server through Socket, and the specific training process is implemented on the server side. Here is a brief introduction to the process of the server using the training samples to train the control strategy model stored in it: In the data collection stage, the server can receive the training samples sent by the client every training cycle, and these training samples will be uniformly stored in the database. The training process of the control strategy model is specifically the process of deep reinforcement learning training. The database used to store training samples is also called the experience replay pool. In the iteration of deep reinforcement learning, when the number of training samples collected in the database reaches a certain number (for example, 2 16 When the deep reinforcement learning agent in the server randomly extracts several times from the database (for example, 2 6 The training samples are grouped into a batch, each of which contains multiple training samples (e.g. 10 The control strategy model is a policy neural network, and the proximal optimization strategy PPO algorithm in the Deepmimic algorithm is used to train the policy neural network. The updated control strategy model has new model parameters, such as the policy weights W0, W1, W2, b0, b1, and b2 in the example of step S502. These weights will be sent to the client in turn, and the client will execute step S505.

[0162] Combined with the above steps, it can be seen that the control strategy model in the client is mainly responsible for forward calculation, that is, calculating the control command according to the association state of the current target virtual character, and the update operation of the control strategy model, such as applying random gradient descent to update the control strategy model, is placed on the server through the GPU. Since the space cost and time cost of using the control strategy model to perform forward calculations are very low, and the time cost of communicating with the server is extremely high, by constructing the control strategy model on the client and transferring the forward calculation function of the control strategy model to the client, the client's dependence on the server and the frequency of downloading control commands from the server can be greatly reduced, thereby improving the efficiency of training.

[0163] In one embodiment, the client sends the sorted training samples to the server via a socket. At this time, there is no need to wait for a reply from the server. The client can use the control strategy model to start the dynamic simulation and data collection of the next training cycle by itself.

[0164] That is: the state generated after the action control strategy of the i+1th training cycle is applied to the target virtual character is collected to obtain the associated state of the target virtual character in the i+1th training cycle; if the model parameters sent by the server are not received within the i+1th training cycle, the associated state of the i+1th training cycle is processed using the control strategy model to obtain the action control strategy of the target virtual character in the i+2th training cycle.

[0165] After obtaining the action control strategy of the i+1th training cycle, the action control strategy is applied in the next training cycle, that is, the i+1th training cycle, to obtain the association state of the target virtual character in the i+1th training cycle. It should be noted that this association state is not the original data after the target virtual character applies the action control strategy, but is obtained after sorting and calculation. The specific process can be referred to the acquisition process of the association state of the i-th training cycle, which will not be repeated here. After the i-th training cycle, the training sample is sent to the server, but before the association state is input into the control strategy model in the i+1th training cycle, the data sent by the server has not been received, and the control strategy model has no new model parameters loaded for updating. Then, the control strategy model for processing the association state of the i-th training cycle is still used to process the association state of the i+1th training cycle. That is, the i-th training cycle and the i+1th training cycle use the same control strategy model to calculate the new association state to obtain a new action control strategy, that is, the action control strategy applied to the i+2th training cycle. And based on the action control strategy in the i+2th training cycle, a dynamic simulation calculation is performed, that is, the torque is calculated.

[0166] Before receiving the model parameters, the control strategy model is not updated, but the client's data collection and motion control strategy calculation for the target control role are still carried out in order according to the sequence of the training cycle. It's just that when the client receives the model parameters, the control strategy model for processing the associated state is different, but it does not affect the client's progress in processing data. In this way, the client can quickly obtain the motion control strategy for dynamic simulation locally, and perform continuous simulation calculations on the action without communicating with the server to obtain the motion control strategy, thus avoiding unnecessary waiting time and allowing for efficient dynamic simulation.

[0167] It is understandable that since the server extracts data for training after the training samples reach a certain amount, the server sends model parameters to the client at a low frequency, and the client does not have to wait for the server model parameters to be sent before it can proceed to the next data processing. In this way, the communication frequency required for data exchange between the server and the client is also extremely low, which can effectively reduce the communication frequency and improve training efficiency.

[0168] S505, receiving the model parameters of the trained control strategy model sent by the server, and updating the control strategy model according to the received model parameters to obtain a target control strategy model.

[0169] In one embodiment, the client updates the control strategy model according to the received model parameters to obtain the target control strategy model. The implementation steps may include: adjusting the model parameters of the control strategy model according to the received model parameters, using the adjusted control strategy model to generate new training samples, and the new training samples are used to continue to adjust the model parameters of the control strategy model; when the adjusted control strategy model meets the model convergence conditions, using the adjusted control strategy model as the target control strategy model.

[0170] The control strategy model deployed or stored in the action simulation engine of the client can be adjusted according to the model parameters sent by the server. The adjusted control strategy model processes the input association state of the target virtual character and outputs the action control strategy. New training samples are organized according to the input-output pairs of the control strategy model and other related data (such as one or more of the association state, action reward, and strategy sampling probability of the next training cycle). The new training samples are sent to the server. When the server collects a certain number of training samples, it extracts training samples to train the control strategy model, and then sends the updated model parameters to the client, continues to adjust the model, process data, etc., and repeats this process until the control strategy model of the client reaches the model convergence condition, and the corresponding adjusted control strategy model can be used as the target model. Among them, the model convergence condition can be that the number of adjustments reaches the preset number, or the action reward reaches the reward threshold, or the change of the model parameters received twice before and after is within the error range, etc., which is not limited here.

[0171] After receiving the model parameters sent by the server, the client will reload these model parameters in the local control strategy model, that is, replace the original model parameters with the model parameters sent by the server, generate a new control strategy model, and generate an action control strategy based on the new control strategy model. The association state of the target virtual character is collected, and the training samples are sorted out according to the association state and sent to the server. This cycle is repeated until a policy performance that meets the task requirements is obtained. The task requirement here can be that the action reward reaches the reward threshold, and the target control strategy model can be obtained. Since the target control strategy model is generated by referring to the target action segment of the target action type, it can well capture the action characteristics of the target action type, and then imitate the action of this action type to generate the corresponding action animation.

[0172] It should be noted that, according to the training steps of steps S501-S505, multiple control strategy models can be trained, each of which belongs to an action type and can generate a control strategy model of that action type when called, and this is caused by the different action types to which the action clips of the training reference belong. In other words, when training, the action clips of a certain action type are input to train the control strategy model, and the control strategy model obtained by training belongs to that action type, and the action animation of that action type can be generated in the application stage.

[0173] In summary, the control strategy model training process in the data processing solution provided in the embodiment of the present application is very efficient. This is because in the training stage, the control strategy model in the client only needs a simple forward calculation, that is, the action control strategy of the next training cycle is calculated according to the associated state of the current training cycle, and the gradient calculation involved in the back-propagation training of the control strategy model, such as the random policy gradient calculation, is performed by the server. The client receives and stores the model parameters sent by the server, and updates the local control strategy model based on the model parameters to achieve synchronization between the trained control strategy model in the server and the control strategy model of the client. Since the time and space costs required for the forward calculation are very low, the model parameters are also sent at a low frequency, and even if the control strategy model of the client is not updated, the corresponding data can still be processed in an orderly manner, and the client's dependence on the server and the frequency of data exchange are greatly reduced. In this way, only a small amount of memory on the client can be sacrificed in exchange for higher training efficiency.

[0174] Combined with the above introduction to the training process of the control strategy model, the following is given: Figure 6 The structure diagram of the control strategy model training framework shown in the figure includes a server and N clients. The server is deployed with a deep reinforcement learning algorithm based on Python language. The same strategy neural network calculation graph, i.e., the control strategy model, is constructed in each client. N clients correspond to N control strategy models. The UE4 engine with dynamic simulation is started in multiple clients (for example, N=1000), and these engines will be started on different CPU cores. Currently, the UE4 editor is generally developed in C++ or C# language, while the deep reinforcement learning algorithm is usually developed in Python language, using Figure 6 The structure shown can realize the data exchange between UE4 and deep reinforcement learning agents.

[0175] The following is an explanation of the specific processing flow involved in the training framework:

[0176] 1) After the last training cycle (e.g. 0.1s) of each UE4 client, the UE4 client can record the current association state of the target virtual character, the target posture of the target action segment (i.e. the action to be tracked in the next training cycle), environmental parameters, current simulation time and other information, and process them immediately. Specifically, these data can be sorted and arranged according to the format of the control strategy model input observation vector defined in the deep reinforcement learning algorithm. The arranged data is forward calculated and randomly sampled by the control strategy model to obtain the action control strategy action (i.e. control command), which is also immediately applied to the control of the UE4 dynamic simulation. At the same time, the client can also calculate the action reward value required for reinforcement learning based on the current state of the target virtual character and the current state of the target action segment.

[0177] 2) The client can upload the observation vector, action vector, the latest state of the target virtual character after the action is executed, the random strategy sampling probability, the reward value, and the current simulation time to the server. At this time, the client does not need to block or wait for the server's reply, and can immediately start the simulation of the next training cycle.

[0178] 3) When the server receives data from any client, it can directly store it in the database. Whenever the number of data in the database reaches a certain value (for example, 2 16 The deep reinforcement learning agent will randomly extract several groups of data from the database according to the principle of stochastic gradient descent to calculate the policy gradient, and use the calculated gradient to update the control policy model through the PPO algorithm. The control policy model updated by the deep reinforcement learning agent is stored on the server for backup.

[0179] 4) After the policy update is completed, the model parameters (such as policy weights) of the control policy model on the server side will be sent to each client. The client loads the updated model parameters and can then continue to efficiently collect data for the next round of updates.

[0180] Repeat the above 1)~4), and after a period of reinforcement learning training, you can get a target control strategy model with good performance, which can stably generate control commands for the target virtual character with dynamic simulation in UE4, guide it to complete the given actions, and generate high-quality animations. The above scheme constructs the same strategy neural network calculation graph in each client. After each deep reinforcement learning agent completes the strategy upgrade, the strategy weight will be sent to each client. Each client can use the current state of the target virtual character to calculate the control command and dynamic simulation locally, and then organize the local data into the format required for deep reinforcement learning and upload it to the server database. During this period, there is no need to wait for the server's response, and data can be collected continuously until the policy weight after the next round of updates on the server is received.

[0181] It can be seen that by executing the forward calculation of the control strategy model on the client, the client's dependence on the server can be greatly reduced, and the frequency of downloading data from the server can be reduced. The calculation of upgrading the strategy using the deep reinforcement learning algorithm is still performed on the server through the GPU, and the server only needs to send the strategy weights at a low frequency. The space cost and time cost of the forward calculation of the strategy are very low, and the time cost of communication is extremely high, so the forward calculation of the strategy can be moved to the client. In this way, only a small amount of running memory on the client (for example, about 10M) needs to be sacrificed, which can greatly save the time required for deep reinforcement learning to collect data. Moreover, the use of the control strategy model deployed locally in UE4 can also greatly improve the computing efficiency of the game when it is running.

[0182] To better illustrate the effect of this solution, please refer to Figure 7 , is a schematic diagram of another training framework of a control strategy model provided in an embodiment of the present application. Figure 7 As shown, it includes N clients and servers. The control strategy model is trained based on deep reinforcement learning. The deep reinforcement learning algorithm based on Python language is placed on the server side. The UE4 engine with dynamic simulation is started on multiple clients (for example, N=1000). These engines will be started on different CPU cores. Figure 7 The data processing flow is as follows:

[0183] 1) Upload status: Within each simulator simulation time period (e.g. 0.1 seconds), each client will compile the current state of the target virtual character, the target posture of the target action segment (that is, the action to be tracked in the next training cycle), environmental parameters, current simulation time and other information into a data packet and send it to the server through Socket.

[0184] 2) Download command: After receiving the data packet, the server will arrange the data in the data packet according to the format of the control strategy model input vector defined in the deep reinforcement learning algorithm. The arranged data will be forward calculated by the control strategy model, and the obtained control command will be sent back to the corresponding client.

[0185] 3) Data storage: The server can calculate the action reward value required for reinforcement learning based on the current state of the target virtual character uploaded by the client and the current state of the known target action fragment, and store the input-output pairs of the control strategy model, random sampling probability, action reward and other information into the database.

[0186] 4) Data extraction and policy update: Whenever the number of data in the database reaches a certain value (for example, 216 groups), the deep reinforcement learning agent will randomly extract several groups of data from the database for policy gradient calculation according to the principle of stochastic gradient descent, and use the calculated gradient to update the policy neural network through the PPO algorithm.

[0187] Repeat the above 1) to 4) and after a period of reinforcement learning training, a target control strategy model with good performance is obtained, which can stably generate control commands for the character with dynamic simulation in UE4, guide it to complete the predetermined actions, and generate high-quality animations.

[0188] The above-described Socket communication solution-based neural network for deep reinforcement learning training provides control strategies for the target virtual character with dynamic simulation in UE4. Since the control strategy model is only stored on the server, the client must communicate with the server once every time step (for example, 0.1s) to upload relevant information such as the status of the current target virtual character, and wait for the server's control strategy model to calculate the strategy and download control commands. This results in a large amount of unnecessary communication delays and waiting when the deep reinforcement learning agent collects data, which not only wastes time, but also introduces new problems for client time synchronization.

[0189] Especially in the process of reinforcement learning, since the data collection process has a strict sequential logic, that is, only after calculating the character state result of the target virtual character in the previous step (that is, the state generated by the control command in the previous step applied to the character), can the control command of this step be calculated, and only after obtaining the control command of this step can the calculation of the next character state be started, that is, the calculation of each step depends on the calculation result of the previous step, so these calculations cannot be parallel. In addition, whenever the client uploads the calculation result of the character state of the target virtual character, UE4 will stop the simulation of the action because it cannot receive new control commands, and wait for the server to receive the state, calculate the control command and send it down before the client can start to continue the simulation. In this way, the time cost is extremely high. For example, if the strategy frequency is 10Hz, that is, the control command needs to be obtained every 0.1 seconds, it needs to communicate with the server 100 times, and it takes about 20ms to use Socket for local communication each time. Then the client actually spends 2s waiting and does nothing at all. It should be noted that when the client simulates a 10-second action clip, without the need for a GUI interface or rendering, and only for data collection, the actual running time is much less than 10 seconds. Therefore, the 2-second waiting time even exceeds the time actually used to calculate the dynamics simulation.

[0190] The solution proposed in the embodiment of the present application adopts the following method: Figure 6 The training framework shown is compared to Figure 7 In this scheme, the client communicates with the server once every time step (0.1s) of simulation, uploads the state of the target virtual character, and waits for the server policy network to calculate and download the control command. This scheme directly builds the policy neural network calculation graph in the action simulation engine in each client (for example, in the UE4 plug-in based on C++), and moves part of the program on the server (that is, the policy network forward calculation action control strategy action) to the client for implementation, thereby achieving the purpose of greatly reducing the communication frequency, greatly improving the training efficiency of deep reinforcement learning, and avoiding the delay of control commands caused by the jam of Socket communication.

[0191] Taking UE4 as an example, this solution can ensure that the control command is calculated within the tick (basic time unit) time of UE4 when the dynamic simulation is turned on in UE4, and complete the high-quality animation with dynamic simulation. By using the UE4 physical simulator to carry out the distributed deep reinforcement learning training framework, the calculation graph is built in each client's UE4 program to store the policy network weights. Figure 7This improvement only sacrifices a small amount of memory on the client (10M level), but can greatly reduce the communication frequency required for data exchange (10 2 ~10 4 times, depending on the task), by reducing the frequency of network communication, it has good real-time performance, high accuracy, and improves the efficiency of training. Using UE4's physical simulator for reinforcement learning training helps to generate complex control strategies, and ultimately can generate high-quality animations with dynamic simulations, which not only reduces game development work and shortens the development process, but also reduces the storage and runtime memory occupied by the game. Under the same conditions, the strategy training time required for the training method provided in the embodiment of the present application is effectively shortened, and the final strategy does not require additional programming, it is automatically compatible with the UE4 program, and is more convenient for game developers to use.

[0192] See also Figure 8 , Figure 8 : is a schematic diagram of the structure of a data processing device provided by an exemplary embodiment of the present application. The above-mentioned data processing device may be a computer program (including program code) running in a computer device, for example, the data processing device is an application software; the data processing device may be used to execute the corresponding steps of the method provided by the embodiment of the present application. Figure 8 As shown, the data processing device 800 may include: an acquisition module 801, a selection module 802, a processing module 803, an output module 804 and a training module 805.

[0193] An acquisition module 801 is used to acquire a user instruction for a target virtual character, where the user instruction includes a target action type;

[0194] A selection module 802 is used to select a target control strategy model belonging to a target action type from a control strategy model library; the target control strategy model is a control strategy model trained based on a target action segment belonging to the target action type;

[0195] The processing module 803 is used to process the state of the target virtual character in the t-th control cycle by using the target control strategy model to obtain the action control strategy of the target virtual character in the t+1-th control cycle, where the action control strategy is used to characterize the action performed by the target virtual character, and t is an integer greater than or equal to 1;

[0196] The processing module 803 is further used to generate the action animation of the target virtual character in the t+1th control cycle and the state of the target virtual character in the t+1th control cycle according to the action control strategy of the t+1th control cycle;

[0197] The output module 804 is used to output the action animation of the target virtual character in all control cycles.

[0198] In one embodiment, the output module 804 is specifically used to: combine the action animations of all control cycles into an animation sequence in the order of recorded generation time from small to large, and store the animation sequence in the animation material library; the action type of the animation sequence belongs to the target action type; when an animation playback instruction containing the target action type is received, the animation sequence is retrieved from the animation material library in response to the animation playback instruction for playback.

[0199] In another embodiment, the output module 804 is further specifically used for: rendering the action animation of the target virtual character in all control cycles respectively according to the set rendering frame rate, and applying the skinning mesh to the rendered action animation to obtain the action animation to be played; the action animation to be played includes multiple animation frames; playing the multiple animation frames in the graphical interface, each of the multiple animation frames is displayed after the synchronization waiting time is reached; the synchronization waiting time is determined according to the set rendering frame rate.

[0200] In one embodiment, the target virtual character includes multiple target rotating objects; the processing module 803 is specifically used to: determine the rotational torque of each target rotating object in the target virtual character according to the state of the tth control cycle and the action control strategy of the t+1th control cycle; according to the rotational torque, control each target rotating object of the target virtual character to move in the t+1th control cycle to generate the action animation of the target virtual character in the i+1th training cycle; after the target virtual character moves, obtain the state of the target virtual character in the t+1th control cycle.

[0201] In one embodiment, the processing module 803 is specifically used to: obtain environmental information and posture information of the target virtual character; the environmental information is used to characterize the virtual environment in which the target virtual character is located; the posture information is used to characterize the position and posture of the target virtual character; and generate the state of the target virtual character in the t+1th control cycle based on the posture information and environmental information.

[0202] In one embodiment, the method is executed by a client, and the data processing device also includes a training module 805, which is used to: obtain the association state of the target virtual character in the i-th training cycle; wherein i is an integer greater than or equal to 1; use the control strategy model to process the association state of the i-th training cycle to obtain the action control strategy of the target virtual character in the i+1-th training cycle, and determine the action reward according to the association state of the i-th training cycle; the control strategy model is stored in the action simulation engine of the client; the association state of the i-th training cycle, the action control strategy of the i+1-th training cycle, the action reward and the association state of the i+1-th training cycle are combined into a training sample; the association state of the i+1-th training cycle is collected after the action control strategy of the i+1-th training cycle is applied to the target virtual character; the training sample is sent to the server so that the server trains the control strategy model stored in the server according to the training sample; the model parameters of the trained control strategy model sent by the server are received, and the control strategy model is updated according to the received model parameters to obtain the target control strategy model.

[0203] In one embodiment, the associated state of the i-th training cycle includes: a first state of the target virtual character, and a second state of the reference virtual character in the target action segment; the first state includes the state of the target virtual character in the i-th training cycle, and the state of the target virtual character in the historical training cycle; the second state includes the state of the reference virtual character in the reference training cycle; the reference training cycle is determined by the i-th training cycle and the set number of training cycles; the state includes position information, posture information and joint angle.

[0204] In one embodiment, the training module 805 is also used to: collect the state generated after the action control strategy of the i+1th training cycle is applied to the target virtual character, and obtain the associated state of the target virtual character in the i+1th training cycle; if the model parameters sent by the server are not received within the i+1th training cycle, use the control strategy model to process the associated state of the i+1th training cycle to obtain the action control strategy of the target virtual character in the i+2th training cycle.

[0205] In one embodiment, the training module 805 is specifically used to: determine the action expectation and action variance of the target virtual character according to the associated state of the target virtual character in the i-th training cycle and the control strategy model; determine the action control strategy distribution of the target virtual character in the i+1-th training cycle according to the action variance and the action expectation; sample the action control strategy distribution in the i+1-th training cycle to obtain the action control strategy of the target virtual character in the i+1-th training cycle; wherein the training sample also includes a strategy sampling probability, and the sampling probability is generated after sampling the action control strategy distribution.

[0206] In one embodiment, the training module 805 is specifically used to: adjust the model parameters of the control strategy model according to the received model parameters, the adjusted control strategy model is used to generate new training samples, and the new training samples are used to continue to adjust the model parameters of the control strategy model; when the adjusted control strategy model meets the model convergence conditions, the adjusted control strategy model is used as the target control strategy model.

[0207] In one embodiment, the training module 805 is specifically used to: perform differential processing on the state of the target virtual character in the i-th training cycle and the state of the reference virtual character of the target action segment in the i-th training cycle to obtain differential information; determine the reward component corresponding to the differential information; the reward component includes a position reward component, a posture reward component and a joint angle reward component; perform weighted summation of the position reward component, the posture reward component and the joint angle reward component to obtain the action reward of the target virtual character.

[0208] It is understandable that the functions of the functional modules of the data processing device described in the embodiment of the present application can be specifically implemented according to the method in the above method embodiment, and the specific implementation process can refer to the relevant description of the above method embodiment, which will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated.

[0209] See also Fig. 9 , Fig. 9 900 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device 900 may include an independent device (such as one or more of a server, a node, a terminal, etc.), or may include components inside an independent device (such as a chip, a software module, or a hardware module, etc.). The computer device 900 may include at least one processor 901 and a communication interface 902. Further, optionally, the computer device 900 may also include at least one memory 903 and a bus 904. The processor 901, the communication interface 902, and the memory 903 are connected via a bus 904.

[0210] Among them, the processor 901 is a module that performs arithmetic operations and / or logical operations, and can specifically be a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a complex programmable logic device (CPLD), a coprocessor (assisting the central processing unit to complete corresponding processing and applications), a microcontroller unit (MCU) and other processing modules, or a combination of multiple of them.

[0211] The communication interface 902 may be used to provide information input or output for at least one processor. And / or, the communication interface 902 may be used to receive data sent externally and / or send data externally, and may be a wired link interface including, for example, an Ethernet cable, or a wireless link (Wi-Fi, Bluetooth, general wireless transmission, vehicle-mounted short-range communication technology, and other short-range wireless communication technologies, etc.) interface.

[0212] The memory 903 is used to provide a storage space, and the storage space can store data such as an operating system and a computer program. The memory 903 can be a random access memory (RAM), a read-only memory (ROM), an erasable programmable read only memory (EPROM), or a portable read-only memory (CD-ROM) or a combination of multiple thereof.

[0213] At least one processor 901 in the computer device 900 is used to call a computer program stored in at least one memory 903 to execute the aforementioned data processing method, such as the aforementioned Figure 2 , Figure 5 The data processing method described in the illustrated embodiment.

[0214] In one possible implementation, the processor 901 in the computer device 900 is used to call a computer program stored in at least one memory 903 to perform the following operations: obtain a user instruction for a target virtual character through a communication interface, the user instruction including a target action type; select a target control strategy model belonging to the target action type from a control strategy model library; the target control strategy model is a control strategy model trained based on a target action segment belonging to the target action type; use the target control strategy model to process the state of the target virtual character in the tth control cycle to obtain the action control strategy of the target virtual character in the t+1th control cycle, the action control strategy is used to characterize the action performed by the target virtual character, and t is an integer greater than or equal to 1; generate an action animation of the target virtual character in the t+1th control cycle and the state of the target virtual character in the t+1th control cycle according to the action control strategy of the t+1th control cycle; and output the action animation of the target virtual character in all control cycles.

[0215] In one embodiment, the processor 901 is specifically used to: combine the action animations of all control cycles into an animation sequence in the order of recorded generation time from small to large, and store the animation sequence in an animation material library; the action type of the animation sequence belongs to the target action type; when an animation playback instruction containing the target action type is received, the animation sequence is retrieved from the animation material library in response to the animation playback instruction for playback.

[0216] In another embodiment, the processor 901 is further specifically used to: render the action animation of the target virtual character in all control cycles respectively according to the set rendering frame rate, and apply the skin mesh to the rendered action animation to obtain the action animation to be played; the action animation to be played includes multiple animation frames; play the multiple animation frames in the graphical interface, and each of the multiple animation frames is displayed after the synchronization waiting time is reached; the synchronization waiting time is determined according to the set rendering frame rate.

[0217] In one embodiment, the target virtual character includes multiple target rotating objects; the processor 901 is specifically used to: determine the rotational torque of each target rotating object in the target virtual character according to the state of the tth control cycle and the action control strategy of the t+1th control cycle; according to the rotational torque, control each target rotating object of the target virtual character to move in the t+1th control cycle to generate the action animation of the target virtual character in the i+1th training cycle; after the target virtual character moves, obtain the state of the target virtual character in the t+1th control cycle.

[0218] In one embodiment, the processor 901 is specifically used to: obtain environmental information and posture information of the target virtual character; the environmental information is used to characterize the virtual environment in which the target virtual character is located; the posture information is used to characterize the position and posture of the target virtual character; and generate the state of the target virtual character in the t+1th control cycle based on the posture information and environmental information.

[0219] In one embodiment, the method is executed by a client, and the processor 901 is used to: obtain the association state of the target virtual character in the i-th training cycle; wherein i is an integer greater than or equal to 1; use the control strategy model to process the association state of the i-th training cycle to obtain the action control strategy of the target virtual character in the i+1-th training cycle, and determine the action reward according to the association state of the i-th training cycle; the control strategy model is stored in the action simulation engine of the client; the association state of the i-th training cycle, the action control strategy of the i+1-th training cycle, the action reward and the association state of the i+1-th training cycle are combined into a training sample; the association state of the i+1-th training cycle is collected after the action control strategy of the i+1-th training cycle is applied to the target virtual character; send the training sample to the server so that the server trains the control strategy model stored in the server according to the training sample; receive the model parameters of the trained control strategy model sent by the server, and update the control strategy model according to the received model parameters to obtain the target control strategy model.

[0220] In one embodiment, the associated state of the i-th training cycle includes: a first state of the target virtual character, and a second state of the reference virtual character in the target action segment; the first state includes the state of the target virtual character in the i-th training cycle, and the state of the target virtual character in the historical training cycle; the second state includes the state of the reference virtual character in the reference training cycle; the reference training cycle is determined by the i-th training cycle and the set number of training cycles; the state includes position information, posture information and joint angle.

[0221] In one embodiment, the processor 901 is further used to: collect the state generated after the action control strategy of the i+1th training cycle is applied to the target virtual character, and obtain the associated state of the target virtual character in the i+1th training cycle; if the model parameters sent by the server are not received within the i+1th training cycle, use the control strategy model to process the associated state of the i+1th training cycle to obtain the action control strategy of the target virtual character in the i+2th training cycle.

[0222] In one embodiment, processor 901 is specifically used to: determine the action expectation and action variance of the target virtual character according to the associated state of the target virtual character in the i-th training cycle and the control strategy model; determine the action control strategy distribution of the target virtual character in the i+1-th training cycle according to the action variance and the action expectation; sample the action control strategy distribution in the i+1-th training cycle to obtain the action control strategy of the target virtual character in the i+1-th training cycle; wherein the training sample also includes a strategy sampling probability, and the sampling probability is generated after sampling the action control strategy distribution.

[0223] In one embodiment, the processor 901 is specifically used to: adjust the model parameters of the control strategy model according to the received model parameters, the adjusted control strategy model is used to generate new training samples, and the new training samples are used to continue to adjust the model parameters of the control strategy model; when the adjusted control strategy model meets the model convergence conditions, the adjusted control strategy model is used as the target control strategy model.

[0224] In one embodiment, processor 901 is specifically used to: perform differential processing on the state of the target virtual character in the i-th training cycle and the state of the reference virtual character of the target action segment in the i-th training cycle to obtain differential information; determine the reward component corresponding to the differential information; the reward component includes a position reward component, a posture reward component and a joint angle reward component; perform weighted summation of the position reward component, the posture reward component and the joint angle reward component to obtain the action reward of the target virtual character.

[0225] It should be understood that the computer device 900 described in the embodiment of the present application can execute the description of the data processing method in the above corresponding embodiment, and can also execute the above Figure 2 or Figure 5 The description of the data processing device 800 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated here either.

[0226] In addition, it should be noted that an exemplary embodiment of the present application also provides a storage medium, in which a computer program of the aforementioned data processing method is stored, and the computer program includes program instructions. When one or more processors load and execute the program instructions, the description of the data processing method in the embodiment can be implemented, which will not be repeated here, and the description of the beneficial effects of adopting the same method will not be repeated here. It is understandable that the program instructions can be deployed on one or multiple computer devices that can communicate with each other for execution.

[0227] The computer-readable storage medium may be the data processing device provided in any of the aforementioned embodiments or the internal storage unit of the computer device, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Further, the computer-readable storage medium may also include both the internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0228] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in the above embodiment.

[0229] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0230] The modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0231] The above disclosure is only part of the embodiments of the present invention, which certainly cannot be used to limit the scope of the rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiments and equivalent changes made according to the claims of the present invention are still within the scope of the invention.

Claims

1. A data processing method, characterized in that: The method is executed by a client, and includes: Acquire a user instruction for a target virtual character, wherein the user instruction includes a target action type; Selecting a target control strategy model belonging to the target action type from a control strategy model library; the target control strategy model is a control strategy model trained based on a target action segment belonging to the target action type; Using the target control strategy model, the state of the target virtual character in the t-th control cycle is processed to obtain the action control strategy of the target virtual character in the t+1-th control cycle, where the action control strategy is used to characterize the action performed by the target virtual character, and t is an integer greater than or equal to 1; Generate, according to the action control strategy of the t+1th control cycle, an action animation of the target virtual character in the t+1th control cycle, and a state of the target virtual character in the t+1th control cycle; Outputting the action animation of the target virtual character in all control cycles; Wherein, the method further comprises: Acquire a training sample, wherein the training sample includes an association state of an i-th training cycle, an action control strategy of an i+1-th training cycle, an action reward, and an association state of an i+1-th training cycle; wherein i is an integer greater than or equal to 1; the action control strategy of the i+1-th training cycle is obtained by processing the association state of the i-th training cycle using a control strategy model; the action reward is determined according to the association state of the i-th training cycle; the association state of the i+1-th training cycle is collected after the action control strategy of the i+1-th training cycle is applied to the target virtual character; Sending the training sample to a server so that the server trains the control strategy model stored in the server according to the training sample; The model parameters of the trained control strategy model sent by the server are received, and the control strategy model is updated according to the received model parameters to obtain the target control strategy model.

2. The method according to claim 1, characterized in that Outputting the action animation of the target virtual character in all control cycles includes: Combining all the action animations of the control cycles into an animation sequence according to the order of the recorded generation time from small to large, and storing the animation sequence in an animation material library; the action type of the animation sequence belongs to the target action type; When an animation play instruction including the target action type is received, the animation sequence is retrieved from the animation material library in response to the animation play instruction for play.

3. The method according to claim 1, characterized in that Outputting the action animation of the target virtual character in all control cycles includes: Rendering the action animations of the target virtual character in all control cycles respectively according to the set rendering frame rate, and applying the skinning mesh to the rendered action animation to obtain the action animation to be played; the action animation to be played includes a plurality of animation frames; The multiple animation frames are played in a graphical interface, and each of the multiple animation frames is displayed after a synchronization waiting time is reached; the synchronization waiting time is determined according to the set rendering frame rate.

4. The method according to any one of claims 1 to 3, characterized in that: The target virtual character includes a plurality of target rotation objects; Generating the action animation of the target virtual character in the t+1th control cycle and the state of the target virtual character in the t+1th control cycle according to the action control strategy of the t+1th control cycle includes: Determining the rotational torque of each target rotating object in the target virtual character according to the state of the t-th control cycle and the action control strategy of the t+1-th control cycle; According to the rotational torque, controlling each target rotating object of the target virtual character to move in the t+1th control cycle to generate an action animation of the target virtual character in the i+1th training cycle; After the target virtual character finishes moving, the state of the target virtual character in the t+1th control cycle is obtained.

5. The method according to claim 4, characterized in that The obtaining of the state of the target virtual character in the t+1th control cycle includes: Acquire environmental information and position information of the target virtual character; the environmental information is used to characterize the virtual environment in which the target virtual character is located; the position information is used to characterize the position and posture of the target virtual character; The state of the target virtual character in the t+1th control cycle is generated according to the posture information and the environmental information.

6. The method according to claim 1, characterized in that The associated state of the i-th training cycle includes: a first state of the target virtual character, and a second state of the reference virtual character in the target action segment; The first state includes the state of the target virtual character in the i-th training cycle and the state of the target virtual character in the historical training cycle; The second state includes the state of the reference virtual character in a reference training cycle; the reference training cycle is determined by the i-th training cycle and a set number of training cycles; The state includes position information, posture information and joint angle.

7. The method according to claim 1 or 6, characterized in that The method further comprises: The state generated after the motion control strategy of the (i+1)th training cycle is applied to the target virtual character is collected to obtain the associated state of the target virtual character in the (i+1)th training cycle; If the model parameters sent by the server are not received in the i+1th training cycle, the control strategy model is used to process the associated state of the i+1th training cycle to obtain the action control strategy of the target virtual character in the i+2th training cycle.

8. The method according to claim 1, characterized in that The process of processing the associated state of the i-th training cycle by using the control strategy model to obtain the action control strategy of the target virtual character in the i+1-th training cycle includes: Determining the action expectation and action variance of the target virtual character according to the association state of the target virtual character in the i-th training cycle and the control strategy model; Determining, according to the action variance and the action expectation, the action control strategy distribution of the target virtual character in the (i+1)th training cycle; Sampling the motion control strategy distribution in the (i+1)th training cycle to obtain the motion control strategy of the target virtual character in the (i+1)th training cycle; The training samples further include strategy sampling probabilities, which are generated after sampling the motion control strategy distribution.

9. The method according to claim 1, characterized in that The updating of the control strategy model according to the received model parameters to obtain the target control strategy model includes: Adjusting the model parameters of the control strategy model according to the received model parameters, the adjusted control strategy model is used to generate new training samples, and the new training samples are used to continue adjusting the model parameters of the control strategy model; When the adjusted control strategy model meets the model convergence condition, the adjusted control strategy model is used as the target control strategy model.

10. The method according to claim 6, characterized in that Determining an action reward according to the associated state of the i-th training cycle includes: Performing differential processing on the state of the target virtual character in the i-th training cycle and the state of the reference virtual character of the target action segment in the i-th training cycle to obtain differential information; Determine a reward component corresponding to the differential information; the reward component includes a position reward component, a posture reward component and a joint angle reward component; The position reward component, the posture reward component and the joint angle reward component are weightedly summed to obtain the action reward of the target virtual character.

11. A data processing device, characterized in that: include: An acquisition module, used to acquire a user instruction for a target virtual character, wherein the user instruction includes a target action type; A selection module, used for selecting a target control strategy model belonging to the target action type from a control strategy model library; the target control strategy model is a control strategy model trained based on a target action segment belonging to the target action type; A processing module, used to process the state of the target virtual character in the t-th control cycle by using the target control strategy model, and obtain the action control strategy of the target virtual character in the t+1-th control cycle, wherein the action control strategy is used to characterize the action performed by the target virtual character, and t is an integer greater than or equal to 1; The processing module is further used to generate the action animation of the target virtual character in the t+1th control cycle and the state of the target virtual character in the t+1th control cycle according to the action control strategy of the t+1th control cycle; An output module, used for outputting the action animation of the target virtual character in all control cycles; A training module is used to: obtain training samples, the training samples include the association state of the i-th training cycle, the action control strategy of the i+1-th training cycle, the action reward and the association state of the i+1-th training cycle; wherein i is an integer greater than or equal to 1; the action control strategy of the i+1-th training cycle is obtained by processing the association state of the i-th training cycle using a control strategy model; the action reward is determined according to the association state of the i-th training cycle; the association state of the i+1-th training cycle is collected after the action control strategy of the i+1-th training cycle is applied to the target virtual character; send the training samples to a server so that the server trains the control strategy model stored in the server according to the training samples; receive model parameters of the trained control strategy model sent by the server, and update the control strategy model according to the received model parameters to obtain the target control strategy model.

12. A computer device, characterized in that: include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide a network communication function, the memory is used to store program code, and the processor is used to call the program code to execute the data processing method described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program executes the data processing method according to any one of claims 1 to 10.

14. A computer program product, characterized in that The computer program product comprises a computer program or computer instructions, and when the computer program or computer instructions are executed by a processor, the steps of the data processing method according to any one of claims 1 to 10 are implemented.